Image-related methods and arrangements

ABSTRACT

A user captures an image of a retail product with a smartphone. Product recommendations associated with the retail product are provided to the smartphone. One claim recites a method comprising: receiving first imagery captured by a smartphone camera, the first imagery representing a first retail product located at a retail location, and presenting the first imagery on a screen of the smartphone; providing the first imagery to a processor to produce fingerprint data therefrom, the fingerprint data being utilized to identify the first retail product; receiving second imagery representing a second retail product, identified as a product recommendation associated with the first retail product, the second imagery being sourced from a source different than the smartphone camera; presenting, on the screen of the smartphone, the second imagery; receiving user input via a touch screen of the smartphone; as a consequence of said user input, initiating an action. Of course, a great variety of other claims, features and arrangements are also detailed.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.14/109,450, filed Dec. 17, 2013 (U.S. Pat. No. 9,595,059), which is acontinuation of U.S. patent application Ser. No. 13/684,093, filed Nov.21, 2013 (U.S. Pat. No. 8,620,021), which claims priority to U.S.Provisional Application Nos. 61/624,037, filed Apr. 13, 2012,61/637,097, filed Apr. 23, 2012, 61/643,084, filed May 4, 2012,61/653,985, filed May 31, 2012, and 61/673,692, filed Jul. 19, 2012. TheSer. No. 13/684,093 application is also a continuation-in-part of U.S.patent application Ser. No. 13/433,870, filed Mar. 29, 2012 (U.S. Pat.No. 9,400,805).

BACKGROUND AND SUMMARY

Social networks are widely used to share information among friends.Increasingly, friends share indications that they “like” particularcontent, such as web pages, songs, etc.

The present disclosure concerns, in some respects, extending concepts ofsocial networks and “liking” to the realm of physical objects (such asmay be encountered in retail stores), through the use of smartphonecameras.

In one particular embodiment, shopper Alice comes across a favoritecookie mix in the supermarket. Her friends raved about the cookies at arecent neighborhood get-together, and she wants to share the secret.With her smartphone, Alice takes a picture of the packaged mix, and anassociated smartphone app gives her the option of “Liking” the producton Facebook.

When she selects this option, the image is analyzed to deriveidentification data (e.g., by extracting an image fingerprint, or bydecoding an invisible digital watermark). This identification data ispassed to a database, which determines the item to which it corresponds.An entry is then made to Alice's Facebook profile, indicating she“Likes” the product (in this case, a package of Bob's Red Mill brandgluten free shortbread cookie mix). A corresponding notation instantlyappears in her friends' Facebook news feeds.

In some arrangements, the app gives the shopper the opportunity toexplore, review, and “like,” related items, such as other products ofthe same brand. For example, by information presented by the app, Alicemay discover that Bob's Red Mill also offers a gluten-free vanilla cakemix. Pleased with her experience with the cookie mix, she decides to trya package of the cake mix for her son's upcoming birthday. Finding itout of stock on the grocery shelf, Alice selects another option on hersmartphone app—electing to purchase the item from Amazon (shipping isfree with her Amazon Prime account).

In another aspect, the present technology is used to share images viasocial networks, such as on Pinterest.

Pinterest is an online service that enables people to share images theyfind on the web. Users compile collections of images (pinboards, orgalleries), which are typically grouped by a common subject (e.g.,vases) or theme (e.g., red items).

In an exemplary scenario, Ted has a fascination for rakes. He has aPinterest pinboard where he collects images depicting the variety ofrakes he's found on the web. While on an errand looking for somethingelse, he happens across a “carpet rake” at the mall department store.Intrigued, he uses his smartphone to snap an image of the barcode labelfound on the product's handle.

A smartphone app gives him an option of posting to his Pinterestaccount. While the barcode, per se, has no appeal, the app automaticallydecodes the barcode and presents a gallery of product photos associatedwith the barcode identifier. Moreover, the app presents images of othercarpet rakes. (Who knew there could be such diversity in carpet rakes?)Ted selects several of the product photos with a few taps, and a momentlater they are all posted to his rakes pinboard on Pinterest.

In a related embodiment, Pat is reading House Beautiful magazine, andsees a picture of a lamp she likes. With her smartphone she captures animage from the magazine page. An app on the smartphone recognizes theimage as having been published in the April, 2012, issue (e.g., by asteganographic watermark in the image), and takes Pat to the HouseBeautiful pinboard for April on Pinterest. There she can click a singlebutton to re-pin the lamp image to one of her own pinboards. Whilethere, she scans the other House Beautiful photos on Pinterest, andpicks a few others for re-pinning too.

The present technology spans a great number of other features andimplementations; the foregoing is just a small sampling.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a smartphone, which can be used inembodiments of the present technology.

FIG. 2 is a diagram of a computing environment in which the presenttechnology can be utilized.

FIGS. 3A and 3B detail one form of user interface that is useful withembodiments of the present technology.

FIGS. 4 and 5 illustrate an embodiment involving print media andincorporating certain aspects of the present technology.

FIG. 6 illustrates aspects of linking images (and text) extracted from acatalog data file.

FIG. 7A shows original text from a catalog, and FIG. 7B shows such textafter processing.

FIGS. 8-10 illustrate aspects of the technology concerning catalogcontent repurposed for social network use.

FIG. 11 shows an incomplete excerpt of an object network.

FIG. 12 shows an excerpt of a data structure relating to the network ofFIG. 11.

FIG. 13 shows a hierarchical arrangement of which the FIG. 11 objectnetwork is a component.

FIGS. 14 and 14A show an illustrative template, and a photo with atemplate feature overlaid.

FIGS. 15A-C, and 16A-C, illustrate certain template-related operations.

FIG. 17 illustrates an embodiment in which still image frames areidentified by reference to audio.

FIGS. 18-21 are flowcharts detailing methods according to certainaspects of the present technology.

FIGS. 22A and 22B are flowcharts of exemplary methods according tocertain aspects of the present technology.

FIG. 23 is an excerpt from an exemplary data structure associatingwatermark payloads with corresponding metadata.

FIGS. 24A and 24B are exemplary data structures used in certainimplementations of the present technology.

FIG. 25 shows a magazine article including a response indicia.

FIGS. 26 and 27 show enlarged response indicia.

FIGS. 28-32 depict screenshots from illustrative embodiments.

DETAILED DESCRIPTION Introduction

The term “social network service” (and the like) is used in thisdisclosure with its ordinary meaning, e.g., an online service, platform,or site that focuses on building and reflecting social networks orsocial relations among people, who share, for example, interests,activities or other affiliation. A social network service typicallyincludes a representation of each user (often a profile), his/her sociallinks, and a variety of additional services. Most contemporary socialnetwork services are web-based and provide means for users to interactover the Internet, such as by public and/or private messaging, and bysharing photos.

Examples of popular social network services include Facebook, Pinterest,Flickr, Google+ and LinkedIn, although different services will doubtlessbecome popular in the future.

Many social networking services provide features allowing users toexpress affinity for certain content (e.g., status updates, comments,photos, links shared by friends, websites and advertisements). OnFacebook, this takes the form of a “Like Button,” which is activated bya user to indicate they “like” associated content. The concept ispresent, with different names, in other social networking sites. Forexample, Google has a “+1” button, and Twitter has a “Follow” button.For expository convenience, this concept is referenced herein by theun-capitalized term “liking.” (As actually manifested on most socialnetworking services, “liking” involves storing—and commonlydisplaying—data reflecting a user's affinity for an item.)

As indicated earlier, implementations of the present technology commonlyinvolve imagery captured by a user's smartphone. FIG. 1 shows a blockdiagram of a representative smartphone, including a camera, a processor,a memory, and a wireless interface.

The camera portion includes a lens, an autofocus mechanism, and a sensor(not particularly shown), which cooperate to provide image datacorresponding to an object imaged by the camera. This image data istypically stored in the smartphone memory. Also stored in the smartphonememory are instructions, including operating system software and appsoftware, which are used to process the image data.

In the depicted smartphone, these software instructions process theimage data to extract or derive image-identifying data from the imagedata. Various such arrangements are known, including digitalwatermarking and image fingerprinting approaches.

Once image-identifying data has been extracted, it is referred to a datastructure (typically a database at a remote server), which uses theextracted data to obtain additional information about the image, orabout the object depicted in the image. If the identification data is anextracted digital watermark payload, it is looked-up in the database toaccess a store of metadata associated with the image/object. If theidentification data is image fingerprint data, a database search isconducted to identify closest-matching reference fingerprint data.Again, based on this operation, a store of metadata associated with theimage/object is accessed. Among the accessed metadata is typically atextual description of the object (e.g., “Bob's Red Mill brand glutenfree shortbread cookie mix”). Additional metadata may include a UPCcode, a product weight, and information about associated online payoffs(i.e., responsive behaviors)—such as a URL that should be followed topresent a related video, etc. In the case of a photograph found in amagazine, the metadata may identify the copyright owner, and specifyprices for different reproduction/use licenses.

In an illustrative embodiment, the smartphone app additionally acts toassociate the object with the user, via a data posted to a socialnetwork service. In particular, the app may upload the image to theuser's Facebook or Pinterest account. Thus, the user-captured image willappear in newsfeeds of the user's Facebook friends, or on a userpinboard published by Pinterest.

As noted earlier, imagery from one product may allow the system toidentify different imagery of the same product, or of different (butrelated) products.

If the user captures an image of a cookie mix in a supermarket, and thesystem identifies the product from its visual features (watermark orfingerprint), the system can use this knowledge to then locatealternative pictures of the same product. These alternate pictures maybe presented to the user on the smartphone—providing the user theopportunity to post one or more of these alternate images to the socialnetworking service (either instead of, or in addition to, theuser-captured image). Thus, instead of a slightly blurry, ill-litsnapshot of cookie mix captured by the user in the grocery aisle, thesocial networking service may instead be provided a marketing photo ofthe product, e.g., from the Bob's Red Mill company's web site (or ahyperlink to such an image).

Knowing the identity of the object photographed by the user, the systemcan similarly identify related objects, and related images. Therelationship can be of various types, e.g., products from the samecompany (e.g., Coke soft drink and Dasani bottled water), similarproducts from different companies (Jimmy Choo biker boots and SteveMadden Banddit boots), etc. Product recommendation software, such as isused by Amazon and other retailers, can be used to identify other itemsthat may be of interest to a person who photographs a particularproduct. (Exemplary recommendation systems are detailed, e.g., inAmazon's U.S. Pat. No. 8,032,506 and in 4-Tell's patent publication20110282821.)

This aspect of the present technology thus can provide the user withimagery, or other information, about these related products. The usermay then elect to post any of this information to their socialnetworking service account (or “like” the depicted items).

FIG. 2 provides a view of a larger system. On the left are the clientdevices (tablets, smartphones, laptops), by which users access theirsocial network accounts, and by which they may take pictures of items ofinterest. These devices connect to the internet, which links them to abank of servers, e.g., at the social network site. These servers containthe programming instructions that implement the social networkfunctionality. These servers also contain the data associated with eachuser's account.

A graphical depiction of Alice's social network account is shown on theright of FIG. 2. Her account comprises a collection of data includingprofile information (name, town, age, gender, etc.), and informationconcerning friends, photos, groups, emails, likes, applications, etc.Much of this data is not literal text (e.g., friends' names), but rathercomprises unique numeric or alphanumeric identifiers (e.g.,39292868552). Each such identifier is associated with various data andproperties (including text and, in the case of pictures, image data).This data is typically stored elsewhere in the social network serverfarm and is accessed, when needed, by use of the unique identifier as anindexing mechanism.

Much of Alice's account data comprises graph information memorializingher relationships to different individuals, websites, and otherentities. In network terminology, the individuals/entities commonly takethe role of network “nodes,” and the relationships between theindividuals/entities (likes, friend, sister, employee, etc.) take therole of “ties” between the nodes. (“Nodes” are sometimes termed“objects,” and “ties” are sometimes termed “links,” “edges,”“connections” or “actions.”) “Liking” an item on Facebook is manifestedby adding a link to the user's network graph, expressing the “like”relationship between the user and a node corresponding to the likeditem. As with nodes, links are assigned unique identifiers, and areassociated with various stored data and properties.

The foregoing is familiar to those skilled in social networkingtechnology. Among such information familiar to these artisans is theFacebook Graph API reference documentation, which is published on theweb, e.g., athttps://developers<dot>facebook<dot>com/docs/reference/api/. (The <dot>convention is used to prevent this information from being rendered as anactive hyperlink when displayed, per Patent Office guidance.)

Magazines, Etc.

Another aspect of the present technology concerns a user who encountersa photograph of interest in a magazine (e.g., National Geographic,Saveur, House Beautiful, Lucky, etc.), and wants to post it to theirPinterest account. Again, the user snaps an image of the magazine pagewith a smartphone, and image processing is applied to identify theimage. This identification may take the form of the magazine name, date,and page number. Or the identification may provide the photographername, and a title or identification code for the photograph in thephotographer's catalog. In either event, the user is presented aweb-stored version of the photograph, and can elect to post it to theirsocial network account.

As before, the identification information allows other, related imagesto be identified. These other images may be related because they areprinted on the same page, or in the same article, or in the samemagazine. Alternatively, they may be related as other views of the samescene/item, etc., or they may depict related items. Still further, theymay be related because they were photographed by the same photographer,or at the same geolocation, as the photo that originally caught theuser's attention. Again, the user can elect to review and—ifdesired—post one or more such related images to their social networkaccount.

In one exemplary embodiment, a user snaps an image of a magazine cover.The particular issue of the magazine is recognized from its visualfeatures, and the smartphone app responds with a user interface (UI)inquiring whether the user is interested in imagery from the editorialcontent (e.g., articles), or from the advertising. If the user respondswith input signaling interest in the editorial content, the app presentsa scrollable list of visual slideshows—each as a horizontal band ofimage thumbnails. Each slideshow corresponds to one of the articles inthe magazine, and includes all the images from the article. (Such a UIis illustrated in FIG. 3A, in the context of the assignee's “Discover”app; the individual thumbnails are not particularly shown.) If the usertaps in any of these horizontal bands, the slideshow expands to fill thescreen (FIG. 3B), and the user can then swipe the display (or touch thearrow icons at the sides) to move forwards and backwards within theslideshow sequence. Each image frame includes a check-box that can betapped by the user to select the image for social network posting.

If the user instead expresses interest in the advertising content, asimilar UI can be used. In this case, the slideshows can be arranged bythe subject matter of the advertisement (e.g., Foods, Travel, Things,Other), by page numbers (e.g., pages 1-20; pages 21-40; pages 113-130),by alphabetic sort of advertiser names, or by any other construct.(Alternatively, the display of multiple bands can be omitted, and asingle, full-size, slideshow encompassing all of the advertisements in adefault order can be presented instead.)

The app's preferences panel, not shown, is used to select the socialnetworking service(s) to which selected images should be posted, and toindicate the default order—if any—by which advertisements should bepresented.

The magazine publisher can facilitate such functionality. For example,the magazine publisher may provide images for an upcoming issue to athird party service provider, who embeds a hidden digital watermark ineach. Each watermark conveys a plural-bit payload, which is stored in adatabase in association with other information. This other informationcan include the name and issue date of the magazine, the page on whichthe photograph is to appear, a publisher-authored caption for thephotograph, and a link pointing to an online version of the photograph.(In some embodiments, the database can actually store image data for thephotograph—either in its original size/resolution as printed in themagazine, or in other formats, e.g., as a thumbnail, or in astandardized size, such as 640 pixels in height. In other embodiments,the online version is stored in an archive maintained by the publisher.)

The digitally-watermarked photographs are electronically transmittedback to the publisher, which prints them in the magazine.

When a user's smartphone later captures an image of one of thesewatermarked pictures from the magazine (e.g., a picture showing adesigner lamp in a lifestyle magazine), a software app on the smartphonedecodes the watermark payload from the captured imagery, and transmitsit to the database. (Alternatively, the decoding can be done by a remoteprocessor, e.g., at the database system, to which the smartphonetransmits image-related data.)

By reference to the received watermark payload, the database retrievesinformation associated with the captured image, and fashions a responseto send back to the smartphone. In the exemplary arrangement, theresponse takes the form of an HTML5 template customized with data fromthe database that defines functionality of several buttons presented onthe smartphone display. The buttons are user-selectable to triggerresponses that correspond to the captured image.

One button, for example, may cause the smartphone browser to load anAmazon web page at which the lamp depicted in the image can bepurchased. Or the app may present links to several vendors that sell thelamp—and display the lamp price at each.

A second button may launch an immersive viewer app or video app by whichthe user can examine imagery of the lamp from all sides. (The SpinCamapp, by SpotMetrix, is one example of a suitable immersive viewer app.)

A third button may cause the system to post a pristine image of the lampto the user's account at Pinterest. This action may invoke another UI,in the app or from Pinterest, allowing the user to specify the pinboardto which the image is to be posted, and allowing the user to type orspeak a caption for the image.

In Pinterest, posting is actually effected by sending Pinterest a URLthat points to a publicly-accessible version of the image, e.g., in anonline version of the magazine, a public archive, or in the databasesystem, rather than sending image data itself. In embodiments employingother social networks, the image data itself may be transferred. (Thisact of posting may be invoked by the smartphone, or by a remotecomputer—such as at the database, depending on implementation.)

A fourth button presented to the user may cause the system to present acollection (e.g., a gallery, or carousel, or slideshow) of relatedimages on the smartphone screen. A scrollable slideshow user interface,such as described above in connection with FIGS. 3A and 3B, is onesuitable arrangement by which these related images can be presented. The3D animated “CoverFlow” (aka fliptych) interface popularized by AppleiTunes and iPod offerings, and detailed in Apple's patent publications20080062141, 20080066016, 20080122796 and 20090002335, is another. Asimple grid layout is still another. In one implementation, the user canselect one or more images from the gallery for posting on Pinterest. Inanother implementation, the user selects one of these related images,and the system then presents a new menu of several buttons (e.g., asdescribed herein), now relating to the just-selected image.

Other buttons presented to the user may be specific to the subjectimage, or may be more general. For example, a fifth button may triggerdownloading of an electronic counterpart of the magazine to thesmartphone's e-book reader, or may download a software app specific tothe magazine or the magazine's publisher.

Of course, other response buttons can be used in other embodiments. Anexample is a Facebook “Like button, or counterpart response buttons forother social networks (e.g., “Follow” on Twitter). Another is a buttonthat triggers an email of the image to user-selected recipients, orposts the image to Facebook or Instagram.

Aspects of such a system are shown in FIGS. 4-5. In FIG. 4, a smartphonereads a watermark from a magazine image, and a responsive action(determined by reference to information in the database system) istriggered. This responsive action can involve retrieving an image froman online archive for posting to Pinterest, purchasing the depictedproduct from Amazon, etc. It may further make use of information storedin the database system

FIG. 5 shows a user's smartphone, in which an HTML5 template downloadedfrom the database system is overlaid on a pristine version of themagazine image captured by the smartphone (again in the context of theassignee's “Discover” app). The template is customized with data fromthe database to define the functionalities of the buttons.

Desirably, the different options presented to the user on the smartphonescreen (e.g., by HTML5-defined buttons) are controlled by the magazinepublisher. This is made possible because the information in the databaseis controlled by the publisher. Thus, the publisher specifies theactions with which the image is associated. This allows the publisher tocontrol ancillary monetization opportunities triggered by the image,such as marketplaces for buying/selling products, attribution, etc.

Attribution refers to giving credit or other value to parties involvedin a value chain. If an image is posted to Pinterest, attribution mayinvolve a caption indicating the image was reproduced with permission ofHouse Beautiful magazine, in which the image appeared in the Apr. 17,2012, print edition. Attribution may also extend to the user who firstcaptured the image from the print magazine, and may include a captionthat this user authored. Such attribution may follow the image whereverit is re-pinned on Pinterest.

Attribution can also involve sharing in monetary or other benefits thatflow from the user act in capturing the image, and the magazine's act inenabling its image to be used in such ways. For example, consider asequence of events that may occur if Ted captures an image of a barcodeon a Carlisle brand carpet rake, and chooses an option from theresulting HTML5 menu that causes a pristine image of such product to beposted to his rakes pinboard at Pinterest. Other users—includingChuck—may re-pin that image from Ted's pinboard to their own pinboards(first-generation re-pinners). Each such re-pinning may include acaption noting the Carlisle brand, and acknowledging Ted as the userthat started the viral spreading of this image. Ted's caption, if any,may also be presented with each re-pinned image, as part of hisattribution. Still other users—including Dave—may find the image onChuck's pinboard, and may re-pin it on their own boards (i.e.,second-generation re-pinners). Again, the Carlisle and Ted attributionscan appear with each such re-pinned image.

If user Ed is intrigued by the carpet rake depicted on Dave's pinboard,Ed can click on it. In response, Pinterest can take one or more actionsand show various links. These links may draw from information in theoriginal database, or from another database maintained by Pinterest(which may replicate some of the data in the original database). Onelink may be to purchase the rake on Amazon. Amazon has a referralprogram, by which parties that direct buyers to Amazon are rewarded witha referral fee. In this case, if Ed purchases the rake from Amazonthrough use of the image depicted on Dave's pinboard, Amazon remits afee for Ed's referral to Pinterest, which may share it with HouseBeautiful, Ted, Chuck, and/or Dave.

In some implementations, the image presented on the smartphone screenfollowing the user's capture of an image from a magazine page is onlybriefly the image captured by the user's smartphone. As soon as thatimage is identified, a pristine image of the file is transmitted to thephone, and presented to the user on the screen—replacing theoriginally-captured image. This image can be part of the HTML5 dataprovided to the phone, or can be delivered separately from the HTML5data. If the HTML5 template includes buttons for user selection, thesecan be overlaid on the pristine image.

In yet other implementations, the identification data discerned from auser-captured image causes the smartphone to launch a Pinterest app (orload a Pinterest web page on the smartphone browser) displaying apinboard on which the magazine publisher has made that image availablefor re-pinning.

Normally, when an image is re-pinned within the digital Pinterest realm,it stays associated with its related metadata—such as the link to itspublic location (e.g., a URL for the House Beautiful image archive), itsattribution information, and any caption. However, if Alice prints afavorite image from Pinterest for posting on her refrigerator, and Bobsees it there and takes a picture of it with his smartphone, Bob's copyof the image is now dis-associated from its Pinterest metadata. (Thesame result occurs if Alice uses her smartphone to show Bob the image ona Pinterest pinboard, and Bob snaps a picture of Alice's phone showingthe picture.)

Desirably, the image captured by Bob′ smartphone can be re-associatedwith its original metadata by reference to watermark data that has beensteganographically encoded into the imagery (i.e., the imagery printedfor display on Alice's refrigerator, or displayed on Alice's smartphonescreen). In particular, a watermark decoder in Bob's smartphone cananalyze the captured imagery for the presence of watermark data. If itfinds a watermark payload conveying a Pinterest image ID (e.g., an 18digit number), it can submit this information to Pinterest to obtain theoriginal URL for the image, together with its attribution information,and present a pristine version of the image—with caption, to Bob. A“Re-pin this image” button can also be presented, allowing Bob to re-pinthe image to one of his own pinboards. Despite leaving the digitalrealm, the image captured by Bob's smartphone has been re-associatedwith its original Pinterest metadata.

The watermark embedding of a Pinterest image ID into imagery can beperformed at the time a user uploads an image for posting to Pinterest.In an exemplary arrangement, a user captures an image with hissmartphone, and launches the Pinterest app. The app includes an optionto upload the image from the user's camera roll (a data structure inwhich smartphone-captured images are stored) to an image repository atPinterest. If the user selects this option, a message is sent toPinterest, together with associated metadata—such as the image size, thedate/time, the user's geolocation information, and the user's Pinterestusername. Pinterest assigns a new Pinterest image ID for the image,stores the just-provided metadata in a new database record, anddownloads to the smartphone a Javascript watermark embedder that isconfigured to embed the assigned ID into an image having the specifiedsize. Using this pre-configured embedder code, the Pinterest appwatermarks the Pinterest image ID into the image, and uploads thewatermarked version of the image to Pinterest for storage. The softwarethen allows the user to pin the image onto one or more pinboards.Thereafter, if the image ever becomes disassociated from the Pinterestecosystem, it can readily be restored by reference to the watermarkedPinterest image ID.

In a related arrangement, the image needn't be uploaded to storage atPinterest. Instead, the user may upload the image to another site—suchas Flickr. Yet the uploaded image is watermarked with the Javascriptcode to include a payload that conveys an image identifier assigned byPinterest. If the image is ever pinned from Flickr to Pinterest, thePinterest database already has information about its provenance.

In still another arrangement, after a user has captured an image from aprint document, and a corresponding online image is located anddisplayed on the user's smartphone, the user taps a button that causes alink to the online image to be posted to the user's Twitter account, orto be sent by SMS service to one or more recipients designated by theuser. The app can automatically include a note, “I saw this image andthought of you,” or the user can author a different transmittal message.This allows users to electronically share their own print-baseddiscoveries.

Print media may commonly have multiple watermarks within a singlepublication (e.g., in different articles, different photographs, etc.).If a reader uses a smartphone to capture one watermark from onephotograph, the backend server that responds to this action can returnresponse data for all of the other watermarked photos in thepublication. This response data is cached in the phone and available tospeed response time, if the user thereafter captures imagery from any ofthose other pictures.

Relatedly, certain magazines may have promotions by which they issuerewards (e.g., $5 Starbucks cards) to readers who scan a thresholdnumber of watermarked images within a single publication (e.g., 10images, all of the Geico advertisements, all of the watermarked images,etc.). The watermark-reading app can give feedback as it tallies thenumber of watermarks read (e.g., “Congratulations, you've scanned 10watermarks. Only 5 to go!”).

When smartphones are used in some environments (e.g., in-flight),network connectivity is not available. In such case, an app may cachewatermark data decoded from print imagery. When network connectivity isthereafter available, the user can recall such information (e.g., from aPending folder) and explore the associated online content.

Another way to handle offline use is for the smartphone app to locallycache, as part of the app software, payoff information that isassociated with different watermark payloads.

Catalogs, Etc.

While this section focuses on catalogs, it should be recognized that themagazine-related arrangements detailed above are also generallyapplicable to catalogs. Similarly, the arrangements detailed in thissection concerning catalogs are also generally applicable to magazines.

A catalog is commonly prepared as a computer data file (e.g., a PDFfile) that includes data elements such as text, photos, and layoutinformation. The file is provided to a printer, which prints a multipagephysical document based on information the data file. The printedcatalogs are then mailed to consumers.

Often, retailers who publish catalogs (e.g., Land's End) also want topublicize their products by posting information on social networkingservices, such as Pinterest. Such posting is presently a highly manual,labor-intensive, process.

In accordance with the present technology, this posting task issimplified. In an illustrative embodiment, the data file is provided toa computer that processes at least some of the data elements to discernseveral themes. The processing also includes associating certain textand photos from the data file with each of these themes. Text and photosassociated with a first theme are then posted to a first online gallery(e.g., a Pinterest pinboard), and text/photos associated with a secondthem are likewise posted to a second online gallery.

This will be clearer with an example. Consider a Land's End catalog. Itsintroductory pages may feature women's wear. A next section may featuremen's wear, and be followed by a kid's wear section. The catalog mayconclude with pages dedicated to bedding.

In this illustrative embodiment, the computer associates each photo inthe catalog with a corresponding text (a “snippet”). This can be assimple as pairing the image on a page with text on the page. (Commonly,catalogs have large images that span a full page, or a two-page spread.)In more complex arrangements, the computer can match letter-keyed textdescriptions (e.g., “D. Madras Shorts . . . ”) with letter-keyed legendsoverlaid on imagery. Additionally, or alternatively, the layoutinformation in the data file can be examined to deduce which text isassociated with which image.

In this particular example, the text on each page is semanticallyanalyzed to produce key terms for clustering. Such analysis commonlydisregards noise words (e.g., “the”, “and,” etc.), and focuses insteadon terms that may be useful in associating the text (and images) withthemes by which pinboards may be analyzed.

(Clustering encompasses a class of computer science operations by whicha set of objects is divided into groups, or clusters, so that theobjects in the same cluster are more similar (in some sense or another)to each other than to those in other clusters. The k-means algorithm(aka Lloyd's algorithm) is commonly used. Clustering is familiar tothose skilled in the fields of data mining and statistical dataanalysis.)

In some embodiments, one or more snippets of text is augmented byadditional terms to aid in the clustering process. For example, analgorithm can examine text snippets for the term “blouse” or “petite,”and, wherever found, add the term “women.” Likewise, the algorithm cansearch snippets for the words “pima” or “worsted” and add the terms“cotton” or “wool,” respectively. Similarly, the terms “trousers” and“chinos” may trigger augmentation by the word “pants.” Such equivalencescan be manually defined by reference to a glossary data set, orautomated arrangements for creating such knowledge bases can be employed(see, e.g., U.S. Pat. No. 7,383,169).

The themes may be prescribed by a human operator, or the themes can beorganically derived by application of a clustering algorithm, e.g.,applied to the key terms. (Clustering based on image features is alsopossible.) In some arrangements, a clustering algorithm examines thetext and suggests several sets of possible themes, between which a humanoperator can then select. Themes such as menswear/kidswear/bedding;belts/purses/socks; and linen/cotton/wool, are examples. In someimplementations, a human operator specifies the number of themes(clusters) desired, and the clustering algorithm responds by proposingone or more divisions of items that yield the requested number ofthemes.

Once the themes are established, the text/image pairings are analyzed toassign each to one or more of the themes. (This analysis commonlyproceeds by reference to text, but alternatively or additionally canproceed by reference to image features.)

An example of a text/image pairing being assigned to plural themes wouldbe where an image depicts both a belt and a purse, and the correspondingtextual descriptions comprise a single snippet. In another example, suchan image may be used in two text/image pairings: one including a textsnippet describing the belt (and assigned to the belt theme) and oneincluding a text snippet describing the purse (assigned to the pursetheme).

Once text/image pairings have been associated with themes, they can beposted to the social network. In the example of Pinterest, each themecorresponds to a different pinboard. If the pinboard doesn't alreadyexist (e.g., Belts), it is created. The pictures are posted, with thecorresponding text snippets submitted as captions for the pictures towhich they correspond. Again, this posting process can be performed by aprogrammed processor, rather than requiring involvement of a humanoperator.

By such an arrangement, data files for print media can be repurposed tocreate marketing tools for social media.

It will be recognized that such process can be performed in advance ofcatalog printing. Alternatively, data files for catalogs previouslyprinted can be so-processed. If any items have been discontinued, theycan be easily deleted from the pinboards.

In some embodiments, such a process is performed by a third partyservice provider. For example, the third party may operate a web sitethrough which retailers can upload their catalog data files forprocessing. The service provider performs the process, and returns“sandbox” data showing the resulting pinboards. This “sandbox” data isnot “live” to the public, but is created for the retailer's approval.Once the retailer approves, data can be sent to the social network tocause the pinboards to go live.

The third party can also—as part of its service—digitally watermark apayload of hidden information into each of the images of the receiveddata file. This typically involves extracting the images from the file,encoding them with a watermark payload, and repackaging the data filewith the encoded images. The third party, or another, can “preflight”the file to ensure that the repackaged file, when rendered into print,still behaves as expected.

The third party can also store metadata in a database that associatesinformation with each of the embedded digital watermark payloads. Theassociated metadata typically includes information such as the retailername, catalog name and date, catalog page on which the image appears,image name, text snippet associated with the image, etc. It may alsoinclude a digital copy of the image—full size and/or thumbnailed.

Returning to the processing of the catalog data file, the computer canalso discern logical linkages between certain of the photos (orphoto/text pairings). This logical linkage information can be used toproduce presentation data that defines certain navigation paths betweenthe images.

Consider shoes. Shoes and other accessories are commonly presented instand-alone images, and are also sometimes presented—incidentally—inimages featuring other items. Thus, a pair of shoes may be among thosefeatured in an image containing only shoes, and the same pair of shoesmay also appear in images that feature jackets and pants.

Such a situation is shown in FIG. 6. These four excerpts are taken fromvarious pages of a Land's End catalog. The processor logically linksthese four spreads, since each depicts the same pair of suede wingtipshoes.

In FIG. 6, two spreads shown to the right (pp. 28-29 and pp. 22-23) eachincludes an explicit text pointer (e.g., “Suede Wingtips, p. 31,” shownin the inset box) directing the reader to the page (depicted at left)where these shoes are featured. The processor discerns the logicallinkage between the images on these pages by commonalty of the text“Suede wingtips” and “31” in all three places.

The same pair of shoes appears in the spread reproduced from pages40-41. Here, however, no explicit text pointer to page 31 is found.Nonetheless, the computer can often discern—by feature matchingtechniques (e.g., by reference to chrominance and texture metrics, andsalient point correspondence) that the shoes depicted in the spread onpages 40-41 are the same as those depicted in the other three locations.

(Note that the spread from pp. 28-29 includes two distinct images—one onpage 28, and the other on page 29. However, since page 28 includes verylittle text, the processor infers that it should be associated with thetext on page 29. The processor will also associate the shoes on page 29with that text due to layout proximity. Based on these circumstances,the process will conclude that a logical linkage also exists between theshoes (on pages 31, 23-23 and 40-41) and the image on page 28.)

The just-noted logical linkages can have different strengths, e.g.,quantified by a numeric scale ranging from 1 to 100. The strength can bea function of several variables. A linkage that is evidenced by a textpointer (e.g., “Suede Wingtips, p. 31”) will be stronger than a linkagethat is inferred by feature matching techniques. Linkages based onfeature matching can be further scored by use of a feature matchingscore. As discussed in the preceding paragraph, the linkage between theshoes and page 28 is derived based on the linkage found to page 29, sothe logical linkages to page 28 will be weaker than the logical linkagesto page 29. In some cases, links may have different strengths indifferent directions. (The strengths of linkages are roughly indicatedgraphically by line weight in FIG. 6.)

As detailed below, these discerned logical linkages can be used, inaccordance with aspects of the present technology, to establish newnavigation routes between the images.

Note that when images used on catalog pages are examined in isolation,they often have large, empty regions on which text is positioned byassociated layout information. The computer processor can sense large,featureless regions (e.g., by small local variance or high frequencymetrics) and crop the image to remove such areas. This is the case withcatalog page 31 of FIG. 6. When this image is posted to the socialnetworking service, a cropped version—such as is shown in FIG. 8, isdesirably used.

Note, too, that the processor desirably parses and edits the catalogtext snippets for posting on the social network. For example, pricingreferences may be semantically detected (e.g., clauses including a “$”symbol or the word “price”) and then removed. Likewise with sizingreferences. Similarly, where the catalog text includes redundancies, theredundancies can be abridged. Where, as in the page 31 example, theimage depicts four shoe styles, the processor may edit the accompanyingtext to produce four snippets—one associated with each shoe style(deleting descriptions of the three other styles), and then post fouridentical pictures to the social network—each with a different caption(corresponding to the different styles).

This is illustrated by FIGS. 7A and 7B. FIG. 7A shows an image of theoriginal text as rendered for printing in the catalog. FIG. 7B shows thetext after processing, for use with the suede wingtips. The processorhas deleted “OUR BETTER PRICE” in the headline, and “FROM $90” in thesubheading. The redundant phrase “NEW! ARCHER FOOTWEAR COLLECTION” atthe lead-in to the descriptive paragraph in the left column has beenomitted (punctuation distinctions, such as “!” are ignored), as has thesizing information at the bottom of this column. The “A” and “B”subparagraphs in the second column, specific to the other shoe styles,have been deleted, as has the catalog number and price of the classicsuede wingtips. Finally, the third column of text—concerning the leatherdriving mocs has been omitted.

FIG. 8 shows the cropped picture, together with the edited text, in thecontext of the Pinterest app (showing “Land's End” as the username, and“Shoes” as the pinboard name).

As is familiar to users, the Pinterest UI shown in FIG. 8 presents thepicture sized so as to fill the width of the screen. If part of thepicture is too large to fit in this presentation, the user can make afinger-swipe-up gesture to scroll-up, revealing the lower part of theimage. The finger-swipe-up gesture also reveals any part of the captionthat doesn't fit in the original placement. (E.g., in FIG. 8, part ofthe caption is off the screen, to the bottom.)

Finger-swiping from side to side does nothing. If the user wishes toreview other pictures on the “Shoes” pinboard, the “Back” button in theupper left of FIG. 8 is touched, which presents thumbnails from the“Shoes” pinboard in three column fashion, which can again be scrolled-upand -down by vertical finger swipes, for use of a next-desired image.(Again, a sideways swipe does nothing.)

In accordance with another aspect of the present technology, othernavigation actions can be taken—based on the linkages earlier discussed.One such arrangement is shown in FIG. 9.

FIG. 9 shows several linking buttons overlaid on the screen. These aresummoned by user command, which can naturally vary by implementation.For example, a double-tap on the picture can cause these buttons toappear. Alternatively, the menu of options that appears responsive to asingle picture touch (which menu presently includes, e.g., “Share onFacebook,” “Save to Camera Roll, etc.), can be augmented to include a“Show Links” button.

Touching any of these links causes the app to present the image (orpinboard) corresponding to the displayed keyword (e.g., Twills). Ifthere are several images (or pinboards) associated with the keyword, theapp navigates to the one with the highest linkage strength (as discussedabove).

Referring back to FIG. 6, it will be recognized that “Twills” button inFIG. 9 will link to the “No Iron Twill Trousers” image (and caption)based on catalog pages 22-23. Similarly, the “Chinos” button will linkto the “Finest Chinos I've Ever Owned” spread based on catalog pages40-41. In like fashion, the “Sportcoats” button links to the spreadbased on pages 28-29.

For clarity of illustration, FIG. 6 does not show a link based on belts.However, it will be recognized that the lower right corner of catalogpage 31 depicts a belt. This belt is recognized by the processor tocorrespond to a two-page spread of belts on pages 42-43. Thus, touchingthe “Belts” button of FIG. 9 will link to this further spread.

The four buttons shown in FIG. 9, and their placements, are exemplary.There can naturally be more or less buttons, and their presentation bythe UI can vary. The depicted buttons are labeled with the names ofproducts (e.g., Twills, Chinos) rather than the class of products (e.g.,Pants), but in other embodiments, class-based labels can naturally beused. If there are more than four links from the image, three linkbuttons can be presented for the strongest links, together with a “More”button. If touched, the “More” button presents menu buttons for anext-strongest group of links.

In some implementations, there is a button for “Shoes”—navigating toanother shoe image/caption that has the greatest link strength to theimage/caption of FIG. 9. (In the illustrated example—in which the imagefrom catalog page 31 is posted four times—each time with a captioncorresponding to a different one of the depicted shoes, the image withthe greatest link-strength will naturally be one of those other threepostings.)

In still other embodiments, arrangements other than menu buttons can beprovided for navigation. For example, finger sweep gestures to the leftand right can cause different modes of navigation. A finger-sweepgesture to the right can lead to the photo with the greatestlink-strength to the base (original) photo, and a finger-sweep gestureto the left can lead to a photo that follows the base photo in theprinted catalog. (If there has already been a sweep to the right orleft, a sweep in the opposite direction simply backtracks. To switchnavigation modes, the user first double-taps the image.)

A variety of such different navigation modes can be implemented, withdifferent swipes leading to navigation in different dimensions ofinformation (e.g., by color direction, by shoe style, etc.).

It will be recognized that some of these navigation acts can lead tophotos posted on pinboards different than the base photo. (E.g., thesuede wingtip shoes may be posted on a Land's End “Shoes” pinboard,while the belts may be posted on a Land's End “Accessories” pinboard.

FIG. 10 shows the smartphone UI after the user has touched the “Twills”button of FIG. 9. The “Twills” image from catalog pages 22-23 isdisplayed, together with the accompanying text caption. The overlaidbuttons remain the same—except the “Twills” button by which the usernavigated from FIG. 9 to FIG. 10 has been replaced by a thumbnail of theshoes image of FIG. 9. Thus, the user can still explore all the linksassociated with the FIG. 9 “shoes” image, even though the imagedepicting twill trousers is now displayed. The thumbnail reminds toserve the user of the base image to which the displayed menu buttonsrelate.

The user can return to the full-size “shoes” image by touching thethumbnail in the upper left.

If the twill trousers now capture the use's interest, the user cansummon link buttons related to the trousers by double-tapping the FIG.10 image, or by touching the image once and selecting an option from amenu—as discussed earlier.

The image/caption pairings, and the relationships between them, comprisea network of objects. FIG. 11 shows a partial view of the network.

In accordance with another aspect of the present technology, networkconstructs used with social networks are here utilized with the presentobject network. For example, as shown in FIG. 11, the network comprisesnodes and links. The nodes typically comprise one or more images and/ortext snippets. Each node is named with a descriptive name—typicallytaken from the text on the page. Thus, on the left of FIG. 11 is a node110 entitled “Archer Shoes.” This node includes an image (here denotedby reference to its page number, e.g., “Image 31”) and a text snippet.The text snippet of node 110 is denoted by “Text 31A”—indicating that itis the first text snippet corresponding to catalog page 31 (of, in thiscase, four text snippets).

Node 110 relates to several other nodes through various links. Each linkis generally named with a relationship class (e.g., “Shoes”), and alsoincludes a strength (not shown in FIG. 11). While only a few nodes areshown, these few nodes link to a much greater number of not-shownnodes—as indicated by the unterminated links.

It will be recognized that certain implementations of the technologydeduce, from the structure of a catalog, a corresponding networktopology. That is, the relationships expressed and implied by thecatalog are mined to determine how the information can be logicallyexpressed in a social network experience. An object graph is synthesizedfrom the catalog to establish a network of things.

In addition to topical links (e.g., “Shoes” and “Pants”), the networkcan include other links. For example, the links in FIG. 11 that areannotated with arrows point to the next page in theas-printed-in-catalog order. (It will be recognized that each such linkcan be traversed in the opposite direction to identify the prior page inthe catalog.) The links with no annotation (e.g., extending betweennotes 114 and 116), are not topically limited, but instead indicate thatthe two objects may be regarded as a unitary object—with topicalcommonality.

The object network of FIG. 11 is implemented, in an illustrativeimplementation, as information in a computer-maintained data structure.The data structure may store identifiers corresponding to the variousimage names, text snippet names, node names, and link names (or thisinformation may be stored directly in the data structure).

FIG. 12 shows one such arrangement, employing a table as the datastructure, with each record (row) in the table corresponding to a nodeor link. Node 110 of FIG. 11 is represented by the number F0001, andnode 112 is represented by the number F0002. Records for these two nodeseach includes the text name of the node, and a file name reference toits data elements (here an image and a text snippet).

FIG. 12 also shows entries in the data structure for two of the links(i.e., the “Belts” and “Shoes” links between nodes 110 and 112). Theformer is represented by the number A0644, and specifies the class ofthe link (“Belts”), the link strength (12), and the two nodes linked bythe Belts relationship (i.e., F0031 and F0022). Similarly for thelatter, “Shoes,” link (represented by the identifier A0823). FIG. 12also shows a data structure entry for a “next page” link between node110 and the following page in the catalog.

(It will be recognized that this simple data structure facilitatesunderstanding of the technology, but a more complex data structure maybe used in actual practice, e.g., a relational database includingvarious additional data elements—such as further object properties.)

The data structure of FIG. 12 is labeled “Land's End, Main, May2012”—indicating its entries correspond to nodes and links found in theMay issue of Land's End's primary catalog. In an illustrativeimplementation, there are other data structures (e.g., tables) for otherpublications.

Consider FIG. 13, which illustrates a small excerpt of the universe ofprint media, in hierarchical fashion. Land's End is one publisher, andit issues a variety of catalogs (e.g., “Outlet,” “Kids,” etc.). Magazineand newspaper publishers also issue a great variety of publications; afew of those from the Conde Nast family of publications are illustrated.

Links can extend between publications. For example, a Land's Endadvertisement for boating moccasins in the May 8, 2012, issue of The NewYorker magazine may be linked to a page in the Land's End June, 2012,“Sailing” catalog. Similarly, a bedspread in the main Land's End catalogmay be linked to a photograph in an article in Lucky magazine where thatbedspread is pictured. Links to other publications can be specified bydata in the data structure, such as prepending “Conde Nast/The NewYorker/5-8-2012/” to a topical link class “Shoes” in the FIG. 12 table.

(If a different table is used to define the object network for eachpublication, then it may be convenient to memorialize links betweenobjects in different publications twice—once in the table for eachpublication. E.g., a link can be expressed by a record in the NewYorker, May 18, 2012, table pointing to an object in the main Land's Endcatalog, and a similar link can be expressed by a record in the mainLand's End catalog table pointing to the New Yorker table. In otherembodiments, object subnetworks for each publication within apublisher's family are stored in a single data structure for thatpublisher, or the subnetworks for each publisher are stored in a globaldata structure encompassing all publishers and their publications. Wherelinks are between two objects stored within the same data structure,then a single link record will commonly suffice.)

Logical links between disparate publications are more difficult todiscern than links within a single publication. However, extension ofthe same techniques can be used. These include text matching, imagefeature matching, explicit references, etc. Google is understood to scanmost magazines (and many other print media) to extract correspondingdigital data from the print media. The data it collects can be processedto discern object links extending between different publications andpublishers.

Like magazines, catalogs are commonly issued on a periodic basis.Although not depicted in FIG. 13, many of the Lands' End catalogs areissued monthly, or every-other month. Desirably, an object in onecatalog that corresponds to the same object in a subsequent catalog isrelated by a link in the Land's End network. (This is shown graphicallyin FIG. 11 by the “Previous Month” link extending from object 110, andby the last record in the table excerpt of FIG. 12.)

Consider what happens if a link is discerned between an advertisementfor suede wingtip shoes in the May 1 issue of The New Yorker, and acorresponding entry in the May 2012 main Land's End catalog. Thenconsider that the May catalog is superseded by a June catalog. In thiscase, the link between the suede wingtip image/text in the May and Junecatalogs can be traversed by a processor that processes data in thenetwork for a user, to identify the June catalog entry as being mostrelevant to that advertisement in the New Yorker (because it is morerecent), even though the June issue had not been published at the timethe May 1 issue of The New Yorker was processed to discern links.

While the foregoing discussion focused on authoring Pinterest-likecollections of imagery, and navigating among the images, it will berecognized that a user can enter this digital experience by capturing animage from a print catalog or magazine (from which image a digitalwatermark is extracted or fingerprint data is computed), as describedearlier. The publisher of such media can arrange for the watermarking ofthe images at pre-press time, or can calculate fingerprint data by whichthe images can be recognized at any time, and store such data (atPinterest or elsewhere) to enable consumers to enter the digitalexperience from the print world.

Text URL- and QR Code-Transcoding

Many magazines and catalogs commonly include text URLs, such as “Formore information, visit www<dot>magazine<dot>com/my-big-story.” Otherspublish blocky QR codes by which readers can link to associatedinformation.

The above-described software tool that takes a PDF publication file, andperforms various processing on it, can also examine the PDF contents forQR codes and for text including URLs (e.g., looking for “www,”<dot>com,<dot>org, etc.). Whenever such indicia is found, the software can applya watermark to that page (or to that portion of the page). The watermarkincludes a payload that is associated, by a backend database, with theURL represented by the text or barcode. (Watermarking of printed textcan be performed in various ways, such as by applying a light yellowwatermark pattern—imperceptible to humans, but machine-readable.) When auser thereafter captures an image of the page with a smartphone, awatermark detector extracts the encoded payload, and links to thecorresponding online destination. (Some versions of this tool mayaugment the existing text of the PDF document to include words like“Scan the article text with your smartphone camera to learn more.”)

Social Network-Based Authoring

In accordance with yet another inventive aspect of the presenttechnology, the contents and/or layout of a publication are determined,at least in part, by reference to information derived from one or moresocial media networks.

Consider Land's End as it prepares its June catalog. The company hascollected sales data for items in its May catalog, so it knows whichproducts sell best. However, it knows relatively little about thecustomers who purchased different products.

Meanwhile, imagery from the May issue has been repurposed by consumerson Pinterest and other social networks. (Land's End may find that manymore people repurpose imagery from the May catalog than actually orderproducts, so the social network use of the catalog can provide a richersource of information than May catalog sales data itself.)

Some of the social network postings (e.g., on Facebook) allow thecompany to discern age, geography, and other demographic data about thepeople who posted catalog imagery. Other postings (e.g., on Facebook andPinterest) allow the company to discern product pairing relationshipsbetween items that were not immediately apparent.

For example, social network analysis may reveal that a knit top with aconservative pattern—which Land's End had targeted for consumers in the45-55 year age bracket, seems most popular with the 18-25 year oldcrowd. For its June catalog, Land's End decides to update the photographof that product to show it being worn by a younger model.

(To explore this issue further, Land's End may decide to run twopictures featuring the knit top in the June issue—one with a youngermodel and one with an older model. After the catalog has been published,the company can analyze the social network repostings of the twodifferent images to gain further data. This analysis can be normalizedto take into account the known age distributions of Pinterest users.)

The company may also find that consumers who post imagery of the knittop from the May catalog to their Pinterest boards frequently (i.e.,more often than random chance would indicate) also post imagery of aparticular pair of canvas slip-on shoes. The association between theknit top and those shoes had not been apparent to Land's End previously,but for the June catalog they decide to follow the implicitly expressedpreference of its social network fans: they decide to picture theseshoes on the same page as the knit top. (Alternatively, the company maydecide to have the photo spread featuring those shoes moved up in thecatalog page order, to immediately follow the page featuring the knittop.)

Although Land's End can survey social networks for information useful inrefining its catalogs, this process may be more efficiently performed bya service provider—such as Axciom—who performs such analyses for avariety of mail order businesses. For each item in a catalog (e.g.,which may be uploaded by the retailer to a computer at the serviceprovider), the service provider can report on the distribution ofinterested consumers based on their demographic profiles. Indeed, theservice provider may be able to match social network profiles withindividual consumers—allowing other consumer-related data in theservice-provider's database to be used in the analysis, and reflected inthe report back to Land's End.

The service provider can also report on the co-occurrence of each itemof catalog merchandise with other items of merchandise—both within thesame user's social network account, and within a particular pinboard inthat user's account. Moreover, the service provider may report on anystatistically significant co-occurrences (i.e., greater than would beexpected by random chance) between postings of particular items ofLand's End merchandise and postings of items from third party vendors.For example, if such analysis shows that a Land's End knit top occurs onPinterest boards to which photos of seersucker shorts from Eddie Bauerare also posted, Land's End may decide to offer a similar pair ofseersucker shorts on the same page as the knit top in its next catalog.

By such techniques, the content and/or layout of catalogs is adapted inaccordance with information gleaned from consumers by their use ofcatalog imagery of social networks.

(While described in the context of catalogs, the same principles can beused in the publishing of books and magazines, in the presentation ofonline merchandise offerings and other information, and in the creationof movies and other entertainment, to tailor the authoring of suchcontent based on social network-based data.)

History-Based Social Network Posting

In accordance with a further inventive aspect of the present technology,a historical log of activity is used in connection with social networks.

Most users who pin new pictures to their Pinterest pinboards (as opposedto re-pinning photos found elsewhere on Pinterest) do so while browsing,by tapping a “Pin” button on a Pinterest toolbar presented by webbrowser software on their computing device (e.g., Internet Explorer 9).

Sometimes, however, people are in a hurry when they browse the web, andthey do not activate the “Pin” button when a desired photo is on thescreen. Also, people who join Pinterest need to start from scratch inlocating favorite photos—no provision is made for photos encounteredbefore joining the social network.

In accordance with a further aspect of the technology, a softwareapplication accesses the “History” file/cache maintained by browsers.The software recalls images from this data store (or re-fetches them, ifthe data store contains only links) and presents the images on the userdevice screen in a gallery presentation. In some embodiments, aPinterest-like pinboard presentation is used. The user scrolls throughthe screens of pictures and simply taps or clicks on the photos ofinterest. The software responds by posting these user-selected photos tothe user's social network (e.g., Pinterest) account.

The software can filter the history by date, so that the user can reviewjust images from web pages visited, e.g., during the past month, orduring June, 2011, or during some other bounded time interval.

In one particular embodiment, the software uploads the first 50 or 100images within the bounded time interval to the user's Pinterest account.The user thereafter uses Pinterest's editing facilities to delete photosthat are not desired (and optionally to move the retained photos todifferent pinboards). This process can be repeated—automatically in someimplementations—for subsequent groupings of 50 or 100 images.

In embodiments of such technology, the software may disregard imagessmaller than a certain size, such as comprising less than 10,000 or20,000 pixels. By this arrangement the user needn't bother with iconsand other small-format image components of web pages, which are unlikelyto be of interest.

While detailed in the context of web browsers, it will be recognizedthat the same principles can likewise be applied to any historydata—regardless of its source. For example, when a user uses a cameraphone to capture an image with the Google Goggles app, and submit it forprocessing, a copy of the image is stored. Such an archive of imagespreviously captured by the smartphone camera can be reviewed, andselected photos can be posted to Pinterest.

Templates

A still further inventive aspect of the present technology involves useof templates with social networks.

In an exemplary embodiment, software on a user device recalls frommemory (or receives from another source) a template data file. Thistemplate data file defines multiple template regions in which differentphotos can be placed. The user selects particular photos for use withinregions of the template, and identifies the placement of each. Theresult is a composite image, which can be posted to a social network.

An illustrative template is shown in FIG. 14. This template is tailoredto aid in creating a pleasing presentation of Hawaiian vacation photos.Some regions of the template are pre-filled with artwork or text. Otherregions are available for placement of images selected by a user from acollection of vacation photos. (The collection may be stored on a userdevice, such as a camera or thumbdrive, or it may be resident in aremote data store, such as the Flickr photo service.)

In some templates, pre-filled elements (e.g., the lei in FIG. 14) can beuser-sized and positioned to overlay a photo in a desired manner—maskingpart of it. Thus, when a user places a photograph of a person in region142, the lei can be made to appear to be around the person's neck—asshown in FIG. 14A.

Drag-and-drop user interface techniques can be employed to aid the userin arranging photos within a template in a desired manner. Aspects ofsuch a user interface are shown in FIGS. 15A-15C. A template with sixregions is shown, populated with six images: A-F. The user can click (ortouch) and drag photo C to the position occupied by photo E. When photoC is released, the user interface automatically snaps photo C to theregion formerly occupied by photo E, and moves photo E to the positionformerly occupied by photo C.

Sometimes the photos being moved, or their respective regions, havedifferent sizes and/or aspect ratios. The software desirably resizesphotos automatically to fill regions of the template in which they areplaced. However, aspect ratios can be handled differently.

FIGS. 16A-16C illustrate this aspect of the technology. Again, the userdrags photo C to the region occupied by photo E. In this case, however,the regions and their new photos have different aspect ratios. In thiscase, pop-up menus are presented—asking whether the user wishes to cropthe edges of the photo to conform to the aspect ratio of the templateregion, or whether the user wishes to maintain the photo's originalaspect ratio.

In the former case, the software removes pixels from the top/bottom (orsides) of the image to fit the region's aspect ratio. This is shown byreformatting of the photo E in FIG. 16C. (In some implementations theuser can adjust the position of the photo within the region to determinewhat part of the image is trimmed.)

In the latter case, where the user chooses to maintain the photo'saspect ratio, the image is re-scaled so that it fits within the region.In such case, a blank or fixed-color region adjoins one or two sides ofthe image. This is shown by reformatting of the photo C in FIG. 16C.(Again, some implementations allow the user to adjust the position ofthe photo within the region, which may result in different-width solidborders added to two opposing sides of the image. In FIG. 16C, forexample, the user has dragged the image to the bottom of its new region,so a black border appears only at the top of this image.)

In still other embodiments, the template is malleable. The user can tapto select a border between regions of the template, and then drag it toadjust the template. This makes one or more of the template areaslarger, and makes one or more other areas smaller.

While templates are usually filed with photos selected by the user, inaccordance with a further feature of the technology, this needn't be thecase. Instead, a template may be prescriptive. It may have a preferencefor what types of images are placed in which regions. Softwareassociated with the template can automatically analyze a collection ofimages specified by a user (e.g., within a computer directory containingHawaiian vacation photos), and populate the template with photosmatching certain rules.

For example, a prescriptive template may specify that region 143 of theFIG. 14 template should be filled with a photo of a flower. The templaterules may further specify that region 144 should be occupied with a shotthat includes a sunset, and region 145 should be filled with a photo ofa seashell.

The software then conducts image analysis of the specified directory ofimages to identify candidate images of each type. Known techniques fordetermining image content are applied. (E.g., sunset is typicallycharacterized by a generally horizontal horizon, with orangish hues inthe top part of the image, and with less luminosity along the bottom ofthe image, etc.) If several qualifying images are identified, thesoftware applies quality metrics (e.g., contrast, color saturation,and/or rules based on artistic composition, etc.) to make a selection.The software may “stack” alternative images in a region, and the usercan review them in sequence by tapping (or clicking) on a region. Eachphoto appears in turn, with the software's top choices appearing first.In some implementations the software has the ability to automaticallycrop photos to zoom-in on a desired element—such as a flower or aseashell, and the thus-processed photo can be inserted in a particularregion.

Typically, the user has the ability to alter the choices made by thesoftware, but the computer-filled template is often a good startingpoint from which the user can then edit.

In some embodiments, the software embeds a digital watermark in each ofthe component images placed in the template. When decoded by readersoftware, the watermark triggers an action—oftenuser-specified—corresponding to that photo. In other embodiments, thecomposite image is encoded with a single watermark—spanning all thephotos. Again, this watermark can be sensed and used to trigger anassociated digital behavior can be launched.

While a template is often filled by an individual, the effort can alsobe collaborative. Two or more people can cooperate, online (e.g., usingtheir smartphones), to create a composite image using a template. Acomputer accessible to both people can host the template, and it canrespond to their respective instructions. The computer can provide turnsto each participant, in a manner familiar from collaborative app games,such as “Words with Friends” and “Draw Something.” In anotherarrangement, the template is transmitted between the users—each takingturns at editing the joint effort. A variety of other collaborativetechniques can also be employed.

Some templates may allow a user to insert a video in certainregions—activated when the user taps or clicks in that region. (Anindicia can be presented in the corner of the region indicating that thedisplayed image is a still frame from a video.)

After the template has been filled, and the composite work has beenfound satisfactory by user-previewing, the template authoring software(or other software) can post the resulting composite work to a socialnetworking service, such as Pinterest. Such an image collection may alsobe designated private, and sent (or a link sent) to particularrecipients identified by the pinboard author.

Desirably, when an image is added to a composite work (e.g., atemplate), it is digitally watermarked with information that enables alater viewer to link back to the original source of that componentimage, or to link to associated content. When the composite work isthereafter shared on social media, individuals can click on the separateimages and be directed to the original image, or to the original uniquepayoffs (e.g., websites, videos, etc.) associated with those photos.

Geographically-Based Posting

In accordance with still another inventive aspect of the presenttechnology, software is provided that enables a user to obtain imageryfrom a particular geography, and post from such software to socialnetworking services.

In one particular implementation, the software invites the user tospecify a geography of interest, such as by name (Statue of Liberty),address (1600 Pennsylvania Ave., Washington D.C.), latitude/longitude,etc. Alternatively, the software can present a user-navigable map (e.g.,such as is provided by Google Maps and Bing Maps). In such arrangements,the user clicks on a desired region, and further navigates by gesturesand on-screen controls to locate an intended geography. (In the case ofa map, the intended geography can span the area displayedon-screen—allowing the user to focus or broaden the inquiry byzooming-in or -out.)

Once the desired geography is established, one or more onlinerepositories of images is searched based on geolocation. As is familiar,this can be done in various ways—such as by text metadata (Statue ofLiberty), zip code, latitude/longitude, etc. The software thendownloads—for presentation to the user—imagery from the specifiedlocale. (The software may screen the imagery to delete substantialduplicates, and present imagery only of a certain quality, e.g., colorimagery having more than 10,000 or 20,000 pixels.)

The images are presented to the user with a UI that facilitates postingto a social network site. For example, the software can provide a “Postto Facebook” or “Pin on Pinterest” control that can be activated, e.g.,by tapping desired photographs, checking check-boxes for desiredphotographs, etc.

Once the user has selected imagery of interest, the software posts theimagery to the desired social networking service.

These principles likewise apply to web sites with geographicassociations. The user may specify a location (e.g., NE Alberta St,Portland, Oreg.), to which the software responds with a Google/Bing mapof the neighborhood. Annotated on the map are attractions, such asrestaurants, etc. If the user clicks (taps) on one of the attractions,the software opens a corresponding website. Alternatively, the softwarecan conduct a search for websites—corresponding to the selectedgeography, and optionally also limited by user-selected criteria (e.g.,“restaurants”). The website then presents imagery from which the usercan select for posting on a social networking service.

By such arrangement, for example, a user at home in St. Louis may browserestaurants in Napa Valley, Calif., and post menu images that appeal, toa “Let's Eat” pinboard on their Pinterest account.

In an alternative implementation, attractions—such as restaurants—canprovide imagery to a user in exchange for receiving something from theuser. For example, the Napa restaurant may provide the user in St. Louiswith access to a gallery of photos showing dishes prepared at therestaurant, in exchange for the user providing information not readilyavailable about themselves. Such information may include their emailaddress, temporary access to their Facebook graph, etc. (Agent softwareassociated with the restaurant's web page can handle such transactionsin automated fashion—including collection of data from the user, andproviding imagery.)

In another particular arrangement, images are provided to a user from anonline repository based on the user's current position (e.g., asindicated by GPS or other location technology). Thus, if wandering inthe menswear section of a department store, the user can review imagesof products that others have posted while in that same section (e.g., asdetermined using geolocation data associated with the prior images).

An app providing such functionality can include a first UI control thatallows the user to specify a distance within which the prior images musthave been posted, e.g., within 15 feet or 75 feet of the user's presentlocation. A second UI control allows the user to specify a time windowwithin which the prior images must have been posted, e.g., within thepast week or month. Images meeting both of these user-set parameters arepresented to the user in an order starting with closest-in-location, andending with most-remote-in-location, irrespective of time.Alternatively, images meeting these parameters are presented to the userwith the most recent first, irrespective of location.

A retailer can make use of such location-based social network postingdata to identify areas of a store where users most commonly seem toengage in social network activity. The retailer can then place signageor other marketing displays in such areas, with the expectation thatusers might post from such displays to social networks.

Pinterest can have a partnership with Foursquare (or otherlocation-based social networking site) by which pinning an image toPinterest from a particular geographical location serves as a “check-in”to Foursquare from that location (helping the user earn points, badges,and other awards). Similarly, to promote use of social networkingin-stores, a retailer can provide incentives (coupons, cash-back, etc.)when a user posts, e.g., 5 or 10 photos captured in a store.

Pinterest can also expose the geolocation data from which users pinphotos, to populate a map showing Pinterest activity, e.g., at differentlocations in a city. A visitor to the city can view the map and click onpins (or thumbnails) representing different Pinterest posts, as a way ofscouting different neighborhoods to see what they have to offer.

Instead of a map view, a smartphone app can present a monocleview—overlaying pins (or thumbnails) on live imagery captured by thesmartphone camera when the user points the phone in different directions(a form of augmented reality)—showing what social network posts peoplehave made in different directions from the user's current location.

Sentiment Surveys

In accordance with yet another inventive aspect of the presenttechnology, software performs a data-mining operation to discernconsumer sentiment from social network postings.

An illustrative embodiment comprises software that crawls publicPinterest pinboards, Facebook pages, or Flickr looking for depictions ofa manufacturer's products. For example, Campbell Soup Company may searchfor depictions of its soup cans.

This effort is aided because many such depictions will be productmarketing photos distributed by Campbell itself. Similarities betweenpictures posted to social networking sites, and reference copies ofCampbell own marketing imagery, can quickly be determined, e.g., usingknown image fingerprinting techniques.

On Pinterest, many such photos will have associated URLs that point backto the web page at Campbell Soup Company on which the original imageappears. Thus, the URLs posted to Pinterest can be crawled, looking for. . . campbell.com . . . in the URL.

User-captured photos of Campbell's products—not originating fromCampbell, can be identified based on known image features—such as acylindrical shape, mostly white on bottom and mostly red on top, etc.Image similarity metrics, such as corresponding SIFT features, can beused for this purpose.

For each posted image, the software harvests any user-authored metadata,e.g., “My husband's favorite” or “Never again.” These annotations arethen semantically analyzed to categorize them into two or more sentimentclassifications (e.g., endorsement, nostalgic, critical, etc.). Astatistical breakdown of such results is then provided back toCampbell's, which can use such information in upcoming marketing andother efforts.

(Sentiment analysis of text is a large and growing field. Exemplarymethods are detailed in patent publications 20120041937, 20110144971,20110131485 and 20060200341.)

Attribution

In accordance with a further inventive aspect of the present technology,digital watermark technology is used to identify a user who first postedan image to a social network, and credit that user for its furtherdistribution on the network.

On Pinterest, when User A re-pins a photo that appears on a pinboard ofUser B, User B is identified on User A's pinboard (in metadata) as thesource. However, if User B repined the photo from user C, user C gets nocredit. Only the immediate “parent” of a pin gets credit—earlier partiesin the content distribution chain are forgotten.

This seems unfair. Much beautiful content is posted on social networks,and the credit for such content should most properly go to the user whofirst introduced it to the network.

In an exemplary embodiment, when a user introduces external imagery to asocial network (as opposed to re-pinning or copying imagery from anotheruser's posting), the image is encoded with a steganographic digitalwatermark. This watermark conveys a plural bit data payload that servesto identify this user. For example, the payload may be a unique 36-bituser number, which is associated with the user's name and otherinformation via a table or other data structure.

When this image is thereafter re-posted (or re-re-posted, etc.) byanother user, the image is analyzed to extract thesteganographically-encoded digital watermark data. By reference to thisdata, the member of the social networking service who first posted thephoto is identified. The social networking service can then publishinformation (e.g., wherever the image is re-posted) giving this originalposter credit for having introduced the photo to the network.

The social networking service can similarly publish a listing of all theusers who re-posted the photo. This may be done, for example, on a webpage associated with the original poster.

Also, the service can publish a ranking of members—showing a number oftimes that their respective originally-posted photos were most oftenre-posted on the social networking service. Members may vie to be amongthe top-ranked entries in such a listing.

Mischief Deterrence

Just as a dissatisfied consumer may establish a gripe web site (e.g.,mitsubishisucks<dot>com), which hosts web content critical of a certaincompany, so too may a user post an image to a social networking site,intending to lead viewers to critical content.

While the web is a good forum for unlimited expression, socialnetworking services—particularly those that rely on advertisingrevenue—may prefer to place some limits on user expression.

Consider the case of a user who is critical of Nike. Such a user maycopy a shoe image from the Nike web site to a gripe site, and then postthe image (i.e., by its gripe site URL) to Pinterest. Users whoencounter the image on Pinterest and click on the image will beredirected to the gripe site (even if the image is removed from the siteafter pinning), instead of to the original Nike site. Nike may take adim view of this, especially if its marketing efforts include a presenceon Pinterest.

To discourage such conduct, Pinterest may check each image newly postedto the service, to see if it matches an image earlier posted. Suchchecking can be done by computing image fingerprint data (e.g., SIFTfeatures) for each new image, and comparing it against fingerprint datafor previously-posted imagery. (A “match” can be more than exactidentity. For example, the image fingerprint data may enable detectionof the same photo at different resolutions, or with differentcoloration, or after cropping or other image editing operations, etc.)

If Pinterest finds that a photo newly submitted for posting correspondsto one already posted, it can then compare the metadata of the twophotos. This metadata may include the associated URLs that point to theweb locations from which they were respectively “pinned.” Based on anoutcome of such comparison, the social networking service can take anaction.

For example, if the service finds a user is posting a new image thatmatches one previously posted, and if the one previously posted has aURL at the nike<dot>com domain but the new one links to a differentsite, the service can amend the URL link of the new image to match theURL of the previously-posted image.

Alternatively, the social networking service may simply send anautomated message (e.g., by email) to Nike alerting it to the posting ofa matching image with a non-Nike URL, and providing Nike with associatedinformation for review.

Still another option is for Pinterest simply to decline to accept thepin. A notification may be sent to the person who attempted the pinning,e.g., reporting that the image should link to the Nike domain.

The action to be taken in a given instance can be determined byreference to rule data, stored in a database by the social networkingservice. The rule data for a particular image may be provided by theproprietor of the web site from which the image was originally pinned(e.g., Nike). Such image proprietor may pay or otherwise reward thesocial networking service for storing and enforcing such rules.

For example, by a web interface, or otherwise, Nike may submit rule datato Pinterest specifying that whenever Pinterest detects a newly pinnedimage that matches an image previously pinned from the nike<dot>comdomain, and the newly pinned image does not also link to thenike<dot>com domain, then metadata of the newly-pinned image should beamended to match the metadata of the previously-pinned image. Campbells,in contrast, may submit rule data simply instructing Pinterest to sendan electronic alert whenever such condition arises for an imageoriginally pinned from the campbells<dot>com web site.

Image watermarking can similarly be used to deter such mischief. Forexample, if Nike wants to ensure that images on its website always linkto its website (i.e., that its images are never repurposed to link toanother domain), it can digitally watermark such images with the payload“nike<dot>com” (or it can watermark the image with a unique alphanumericidentifier that resolves to “nike<dot>com” in a watermark database).Pinterest can check all images pinned to its social network site forembedded digital watermarks. When it finds an embedded watermark payload(e.g., “nike<dot>com”), it can check that the URL associated with thislink is at the Nike domain. If not, it can decline to accept the pin, ortake other responsive action, as detailed above.

Such arrangements are useful to ensure that images for branded productsalways link back to their respective brand owners.

Posting Images from Video

Audio information can be used in posting images from video to socialnetworking sites.

Consider a user who is watching Saturday Night Live, and wants to postan image from a skit to the user's Pinterest site. Using a smartphone,the user captures audio from the program. A processor—in the phone or ata remote site—processes the captured audio to derive identification datafrom it, such as by digital watermark decoding or audio fingerprinting.

This identification data is used to access a store of content related tothat program. This store may include an ISAN content identifier, and mayalso include a pointer to an online gallery of still image frames, e.g.,provided by the producer of the television program for marketingpurposes. (Or, a different database accessed using the ISAN identifiermay include such a pointer.) The user reviews the gallery of marketingimages—on the smartphone screen or on another screen—and pins one ormore desired images from this gallery to Pinterest (i.e., sending a URLfor the desired image—identifying its location in the online gallery—toPinterest).

FIG. 17 illustrates such a method.

Audio Accompaniment

Relatedly, images posted to social networking services can be associatedwith corresponding audio or video clips. When a user taps such an imageon a smartphone (or hovers over such an image with a mouse on a desktopcomputer), identifying information is extracted from the image data.(Again, digital watermark decoding or image fingerprinting techniquescan be used.) This identifying information is used to access a datastore in which content related to the image is stored. This content mayinclude audio or video information, or a link to same. Such audio/videocontent is rendered to the user—either automatically, or in response toa further user instruction.

In some embodiments, a short (5-10 second) snippet of compressed lowbandwidth audio (e.g., 3 Khz) is steganographically encoded into animage (e.g., by a fragile watermarking technique, such as least bitsubstitution). When the user taps or hovers over the image, the audio isdecoded and rendered directly from the image data.

By such arrangements, a user can, e.g., annotate a Pinterest image postwith commentary about the subject depicted in the image, or captureconcert audio to accompany a photo of a band performing.

Use in Retail Stores

The present technology finds many applications in retail settings.

One arrangement makes use of signage at a store (printed, or displayedon an electronic screen), depicting a product offered for sale. With acamera of a portable device, a user captures a photo of the sign.Identifying information (e.g., watermark or fingerprint) is thenextracted from the captured image data.

The identification information is then used to access locationinformation for the depicted product within the store. For example, thisinformation can reside in a database maintained by the store, or a moreglobal database—serving many different stores—can be employed.

The information accessed from the database can also include navigationinstructions to guide the user from the sign to the product location(e.g., using turn by turn directions leading the user through the storeaisles, and/or a store layout map with one or more arrowsoverlaid—depicting the route). Or, such instructions can be computeddynamically—based on the user's present location (as sensed by softwarein the user's device), using known indoor (in-store) navigation softwaretools.

Another arrangement makes use of images previously posted to a user'ssocial networking site, to discern user interests. Alerts can then beprovided to the user based on nearby products.

Consider a user who posted an image of Jimmy Choo motorcycle boots toher Pinterest page. When the user thereafter is near a retailestablishment that stocks these boots, a push notification may be sentto the user's phone—alerting her to the product, and the store location.

The correspondence between the boot image posted to the user's socialnetwork account, and the store product, can be discerned in variousways. One case is where the user has pinned the image from the store'sweb site. For example, if posted from the Nordstrom site, the post maycomprise the Nordstrom URL:shop<dot>Nordstrom<dot>com/s/jimmy-choo-motorcycle-boot/3069637?cm_ven=pinterest&cm_cat=pinit&cm_pla=site&cm_ite=3069637.

Software on the user's phone can maintain a list of the domains fromwhich images are pinned (e.g., target<dot>com, nordstrom<dot>com, etc.)and can periodically check whether the user is near any of these stores.(“Near” is a parameter that can be user-set, e.g., within a mile, within100 yards, within 10 yards, within wireless range, in-store, etc.) Thepresence of such a store can be determined by reference to Google Mapsor the like, through which the user's location on a map can be comparedwith known locations of different stores. Another alternative is forstores to send out wireless data (e.g., a WiFi network naming the store,Bluetooth, or ultrasonic audio) announcing themselves. Still otherimplementations can involve the user device periodically updating a webservice with the device location. The web service can then determineproximity to different stores, and may also have knowledge of the webdomains from which the user has posted images, so that nearness to oneof those stores can be determined.

Once the user is found to be close to a Nordstrom store, the user devicecan send the image URL to the store, e.g., by Bluetooth, WiFi, etc. Thestore applies the received URL to a data structure that maps differentweb page URLs to the corresponding SKUs or UPCs (or other in-storeidentifiers) for those products. The store then checks its electronicinventory records to determine whether the item indicated by thereceived URL is in-stock. If so, an alert is sent to the user device,for display to the user on the device screen.

Instead of being posted to the user's social networking site by adomain-specific URL, an image may also be posted by reference to a UPCidentifier (e.g., decoded from a barcode or other indicia). The userdevice may periodically broadcast—by WiFi, Bluetooth, etc., a list ofUPC identifiers for items depicted on the user's social networking site.Nearby stores that receive such broadcast can check their inventory todetermine whether any of the thus-identified products is in theirinventory. If so, an alert can again be transmitted for display to theuser.

Instead of broadcasting such a volume of data, the collection of UPCidentifiers can be stored in an online repository associated with theuser. The user device can simply periodically broadcast an identifier bywhich stores (or associated web services) can access this repository.Nearby stores that receive such broadcast can access the online listusing the broadcast identifier, and alert the user if any match isfound.

In some embodiments, descriptive text associated with an image postingis used to identify a product of interest to the user. For example, fromthe Nordstrom URL given above, the text Jimmy Choo Motorcycle Boot canbe extracted. Many URLs similarly include semantic text (e.g., mostAmazon URLs incorporate such text). Such text-based product descriptorscan be periodically broadcast from a user device (or an identifier of anonline repository where such descriptors is stored can be broadcast),and nearby stores can check their inventory to determine whether anyproduct having such a descriptor is in-stock. Again, correspondingalerts can be issued.

In other cases, the URL itself does not include descriptive text.However, such descriptive text can be extracted from the web page towhich such a URL points. (This is particularly the case if Web 2.0technologies are used—labeling the different web page components, e.g.,including ProductName.)

Other embodiments use image fingerprint or watermark data. That is,fingerprint/watermark data for images posted to a user's socialnetworking service can be computed and then broadcast (or stored in anonline repository, accessed by an identifier that is periodicallybroadcast). Nearby stores can compare these identifiers withfingerprint/watermark identifiers for reference photos depictingin-stock products. Again, if any match is found, the user is notified.

Given one type of identifier (e.g., a URL, a UPC code, image fingerprintdata, watermark data, descriptive text, etc.), an online data store(e.g., a translation table) can provide one or more correspondingidentifiers of other types (URL/UPC/fingerprint/watermark/text, etc.),which can be used in the presently-detailed arrangements. Suchtranslation tables can be provided by individual retailers, or anindependent operator can host such a service for a larger variety ofproducts.

In embodiments in which the user device periodically broadcastsinformation to nearby stores, battery life can be preserved by actuatingsuch functionality only when the device location is determined (e.g., byGPS or the like) to be within an area having a store nearby (or having aconcentration of multiple stores within a small region, such as 10 or 50stores within 1000 feet—such as in a shopping mall). Google Maps andother online services can provide such information about the locationsof stores.

In the foregoing arrangements, an alert can also (or alternatively) beissued when the user is near a store that stocks a product depicted inan image posted by a social network “friend” of the user. This canfacilitate shopping, e.g., for birthday gifts.

The alert provided to the user can include the photo of the object fromthe social networking site.

In a variant arrangement, stores or malls can offer a shopper conciergeservice. A shopper sends a link to one or more pinboards, and anemployee researches the availability of the depicted merchandise at thatstore/mall. When the shopper arrives, the concierge greets them andaccompanies them directly to the merchandise depicted on thepinboard(s).

In still other arrangements, a store sends a gallery of photos to anearby user. Included first in the gallery are any in-stock items thatare among those pictured in the user's social networking sites. Next arepresented any such items that are among those posted by the user'sfriends. Finally, the store may include photos of items that are mostfrequently posted by other social networking site users (optionally,those with demographic profiles most similar to that of the user).

In some implementations, the user needn't be physically near a store totake advantage of such functionality. Instead, the user can placethemselves virtually at different locations, to explore differentshopping opportunities. For example, if a user plans a Christmas trip toNew York, the user can virtually place themselves at different locationsin the shopping district (e.g., by a UI that allows entry of a remotelocation), and see what products depicted on their social networkingsite are available, and where.

The above-detailed functionality can be integrated into a software appoffered by a store (e.g., Nordstrom), or tools generic to differentstores can be employed.

Sometimes a store may not have photos of all the products stocked on itsshelves. Or a store may wish to author a Pinterest page showing customerfavorite products—with a minimum of effort. To fill such needs, thestore may rely on crowdsourcing, i.e., photos captured by shoppers.

For example, a retailer may search online photo collections (e.g.,Flickr, Pinterest, Facebook, etc.) for photos that have accompanyingmetadata indicating the photos were captured within their store. Thismetadata may comprise, e.g., latitude/longitude data that was stored bythe user device at the time of image capture, or it may comprise a textannotation provided by the user (e.g., “Saw these shoes at the downtownPortland Nordstrom store.”) Such images can then be re-purposed by thestore, such as re-pinning onto the store's Pinterest page. (Known facialdetection techniques can be applied to such imagery before suchre-purposing, to ensure that no recognizable individual is present inany such photo.)

Alternatively, a store's WiFi network can use packet sniffing to detectimage traffic sent from within the store to Pinterest. When encountered,it may copy such images and re-purpose them for its own use. (Naturally,such sniffing and repurposing should only be employed where expresslyauthorized by the user, such as in terms of use to which the user agreedbefore using the retailer's WiFi network.)

Related arrangements can be employed by public attractions other thanstores. For example, the city of Portland, Oreg. may compile a Pinterestpinboard showing photos of its city parks, by reviewing public photopostings depicting imagery captured within geographical boundaries ofthe city's parks.

Some retailers may create a Pinterest board for each aisle in theirstores—showing photos captured by shoppers while in those respectiveaisles. (Indoor positioning technology with resolution fine enough toidentify a user's location by aisle is known, e.g., as detailed in U.S.Pat. Nos. 7,876,266 and 7,983,185, and published US Patent Application20090213828.) Or even finer geographical or topical granularity can beemployed, e.g., cotton sweaters on one pinboard, neckties on another,etc.

While most of these embodiments involve user interaction withsmartphones, other arrangements can alternatively (or additionally)involve user interaction with public electronic displays. For example, atouch-screen display panel at the entrance to a Nordstrom store, or atthe entrance to a mall, can display images (which may be posted to andpresented from one or more Pinterest pinboards), showing popularproducts.

Popularity can be judged in various ways. For example, cash registersales data from the past week can identify products that have had thehighest unit or dollar sales, or the display can show products availablein the store/mall that have most often been posted to Pinterest. Stillanother arrangement shows a scrolling timeline ticker ofimages—depicting each item in the store/mall as it is purchased, orposted to a social network.

Shoppers can interact with the electronic display by tapping photos ofinterest. While many different responses are possible, one responds byidentifying the product by name, price, and location in the store/mall.Additionally or alternatively, the tapped image may be re-pinned to theshopper's Pinterest board.

To enable re-pinning of publicly-displayed images to the user's account,the user may be presented with a login screen to enter Pinterest log-incredentials. Perhaps preferable is for such information to beautomatically and wirelessly conveyed from the user's smartphone to theelectronic display system—in accordance with user-set rules aboutconditions in which such data can be shared.

Still better is for the tapped image to be re-pinned to the shopper'sPinterest account without providing any credentials to the displaysystem. Instead, the app on the shopper's smartphone can send dataindicating the phone's location to Pinterest (or an associated webservice), which uses this information to identify a public displaynearest to the shopper. Pinterest then asks the display to send it dataindicating the image that was most-recently tapped. Pinterest thenre-pins that image to the user's account. (The smartphone app may firstdisplay the image that Pinterest deduced was of interest to the shopper,inviting user confirmation before it is pinned.)

In yet another arrangement, Pinterest takes the location informationmade available from the phone, and provides the smartphone app with acopy of the pinboard information being displayed on the nearest publicelectronic display (re-formatted for presentation on the smartphonedisplay). The user can then scroll and select from among these imagesfor re-pinning. In some embodiments, a dynamic stream of images ispresented on the smartphone—corresponding to changing imagery presentedon the public display. If the user moves from that location, the feed ofimagery may continue. (While taking a break at a mall coffee shop, theuser may thereby review what items are being sold, e.g., at Nordstrom.)

In other arrangements, pinboard data is made available to the user'ssmartphone only so long as the user is in proximity to the publicdisplay (or other location with which a pinboard is associated). Thedata provider (e.g., Pinterest) may require the app to send it currentlocation data every minute, and if none is received (or if the dataindicates the user has moved outside of a zone associated with aparticular pinboard), then no new data is provided.

(Still another option for re-pinning an image shown on a public displayis to use the techniques detailed earlier, e.g., based on capturing thedisplayed image with the smartphone camera, and recognizing same byimage fingerprinting or watermarking.)

Social Discovery

Imagery can provide a means for discovering users with similarinterests.

Consider a first user who takes a photo of a particular book while at abookstore, and posts it to the user's social network account. The socialnetwork service analyzes the image (e.g., by calculating fingerprintdata) and matches it to two other images previously posted to the socialnetworking service by two other users. The social network service canthen notify the two previous users about the first user's new post aboutthe same book, and can likewise alert the first user to the two previoususers' posts (all subject to privacy safeguards, such as opting-in tosuch service). This notification may comprise only the photo(s) taken bythe other user(s), or it may comprise more extensive information—such asother photos posted by such user(s).

In some embodiments, such notifications only occur if the photos werecaptured at the same location (e.g., at the same bookstore).

Pinterest-to-Print

Images from Pinterest can be used to compile print documents—such ascards and booklets. For example, a user's friend may be a Ducatimotorcycle enthusiast. The user can compile a pinboard of Ducatimotorcycle images, and use a printing option on Pinterest (or a thirdparty service provider) to produce a bound booklet of such photos, whichis then mailed to the user (or is mailed to the friend, or is held forpickup at a nearby print shop).

Each image in the booklet can be watermarked. If the friend wishes tolearn more about any of the depicted Ducati motorcycles, she can captureimagery from the printed booklet with a smartphone app that decodes thewatermark information, and leads to the web site on which the image ishosted.

Similarly, a user who is on vacation in Europe can browse Pinterest forimages related to his trip, and direct that they be printed and mailedas postcards from the United States, with personalized messages.

Reconciling Resolutions

Situations can arise in which different parties may want a decodedwatermark (or other recognized content) to trigger different payoffs.For example, one party may be a publisher of a watermarkedadvertisement, while the other party may be a distributor of awatermark-reading app for smartphones. Whose payoff preference shouldprevail?

Consider EBay. It may offer a smartphone app that reads watermarks. (Italready offers such an app for reading barcodes.) When it reads awatermark from a magazine ad for a Rolex watch, the decoded payload maypoint to a record in a backend database that identifies the watch bymodel number, and provides a URL to a Rolex web site that identifiesauthorized resellers. The EBay app may disregard the URL information,and instead use the model number information to link to an EBay web pagepresenting dozens of such Rolex watches for sale.

The advertiser, Rolex, may take a dim view of this. Having paid for theadvertisement, and having taken the effort to provide a watermark, itmay want its specified URL used as the payoff, so that viewers of itsmagazine ad are provided information about authorized resellers.

As another example, consider Amazon. Sometimes it may partner with brandowners to promote sales through Amazon's web site. It may offerincentives, for example, for the watermark used in a Wilson SportingGoods ad to point to a backend database record having a URL identifyingan Amazon page from which the advertised Wilson product can bepurchased. The URL identified by a watermark in an advertisement forWilson's “Blacktop Warrior” basketball may thus bewww<dot>amazon<dot>com/Wilson-Blacktop-Warrior-Basketball-Orange/dp/B001URVJ0W/ref=sr_1_8?

An EBay app encountering such advertising may access the database recordand obtain the URL data. Instead of using the URL to link to Amazon,however, the EBay app may extract semantic information from the URL(i.e., Wilson Blacktop Warrior Basketball Orange) and repurpose thisinformation to link to EBay web pages that offer the same Wilsonbasketball for sale.

More generally, there may be multiple sources of preference information,e.g.: (1) the party that encodes the watermark, (2) the app that readsthe watermark, and (3) the consumer who uses the app. The followingdisclosure details a few of various ways that the preferences of thevarious parties can be reconciled.

In one particular embodiment, watermark payloads may have associatedPurchaseAttribute data. This data can form part of the payloadinformation that is literally conveyed by the watermark in the mediaobject or, more commonly, this data is stored in a database recordindicated by the encoded payload. The PurchaseAttribute data indicatesthe preference of the party responsible for the watermark—indicating howany purchase initiated from the watermark should be fulfilled.

FIGS. 22A and 22B illustrate how watermark information decoded from amagazine advertisement is handled differently by two differentsmartphone apps. One is an EBay app that is specialized for purchasingproducts from EBay. When it is launched, the app presents a splashscreen with the name “EBay” (which is stored with other programinformation), and launches a camera application to capture imagery. Theother app is similar, but not associated with a particular retailer. TheDigimarc Discover app is one such example.

Both applications start by processing imagery captured by the smartphonecamera (e.g., from a magazine page or product packaging) to decode theplural-bit watermark payload. They then access information in thedatabase record that corresponds to this payload. PurchaseAttribute datamay be among the data stored in the database record and encountered bythe apps.

If PurchaseAttribute data is present, the FIG. 22A EBay app next checkswhether this data includes the string “EBay” (or other EBay-identifyinginformation). If so, the application branches to a routine (notparticularly detailed) for purchasing the product, employing EBaysign-in credentials previously stored by the user.

If, in contrast, PurchaseAttribute data is present, but doesn't includeEBay-identifying information (e.g., it includes “Amazon”) then the EBayapp understands that it should not invoke the routine to purchase theproduct through EBay. Instead, the application branches to a gracefulexit.

The graceful exit may not require the EBay application to terminate.Instead—functionality other than purchasing on EBay may be pursued. Forexample, the EBay application may provide information about theproduct—such as from a manufacturer's web site. Alternatively, the EBayapplication may provide the user with information about completed EBaytransactions involving the product—indicating the prices paid by others.Another alternative is for the EBay application to provide arecommendation of another application that is preferred for use withthat watermark (per application-identifying data provided from thedatabase). The EBay app may even provide a link that the user can followto Amazon, although this would be unusual.

Sometimes the watermark-associated database record may not have anyPurchaseAttribute data. In this case, the EBay application (FIG. 22A)may invite the user to purchase the subject item on EBay.

If a watermark is read from editorial content (such as a nationalGeographic article) a related, purchasable product may be derived by theEBay app from the information returned by the database. For example, ifthe article concerns Hawaiian history, EBay might provide links toHawaiian vacation packages, Hawaiian art, etc.

Similarly with novelties such as a baseball trading card. One link maybe to a video about the baseball player. Another link presented by theEBay app may be to memorabilia involving that player, for sale on EBay.

The database record may include data indicating whether the itemassociated with the watermark is purchasable. Alternatively, thewatermark payload itself may reveal this information. In a particularembodiment, the latter approach is used. If the watermark payload is aneven number (e.g., in hexadecimal), this indicates the item ispurchasable. If odd, then not.

Another approach is with payload versioning. The watermark payload maybe extensible, so that it can be extended to accommodate additionalinformation as-needed.

In the case of a National Geographic article, the graceful exit shown atthe left side of FIG. 22A can comprise the EBay application linking to aNational Geographic web site associated with the article.

FIG. 23 details an excerpt of a database to which the smartphone appslink when they encounter a watermark. The first entry is the watermarkpayload. This is the index by which the app identifies the record (row)in the database corresponding to the just-read watermark payload. Thenext entry is the PurchaseAttribute data, specifying an authorizedvendor(s) of the item associated with the watermark payload.

Next is name text associated with the watermark payload. This text maybe presented to the user as part of the app's response to reading awatermark. Also provided is a class of the item (e.g., electronics,books, movies, groceries, household, etc.) Following that is an entrycontaining a product identifier—such as an EAN or UPC number.

The database record further includes a web link that can be used by theapp to present more information about the item associated with thewatermark. The next entry in the row is a URL that can be used for theapp to present a purchase opportunity to the user. (In someimplementations, this web link serves as the PurchaseAttribute data,since it typically includes the name of the authorized vendor(s).

The database record may also comprise further information, as may bedictated or desired by the particular implementation. For example, thedatabase may include a reference to another database that the phone appcan query for additional information. Additionally, as noted earlier,the application may present a variety of different links that the usercan pursue—instead of just the one or two links per payload shown inFIG. 23.

(In some implementations, the database record corresponding to thewatermark may be sent from the database to the smartphone, in responseto a smartphone query.)

Returning to FIG. 22A, if the EBay app decodes the watermark payload43DF8A from packaging for Sony headphones, it will find—from the firstrow in FIG. 23—that EBay is not a permitted vendor for this item.(Amazon is specified.) Accordingly, the flow chart of FIG. 22A indicatesthat the app will branch to a graceful exit, rather than continuetowards offering the user a purchase opportunity on EBay.

If the EBay app decodes the watermark B163A4 from a magazineadvertisement for Kleenex tissues, it finds—from the third row in theFIG. 23 database—that there is not a PurchaseAttribute set for suchitem. Since the payload (B163A4) is an even number, the app understandsthat the item is a purchasable product. So the app follows the branch tothe bottom of the FIG. 22A flow chart—proceeding to present a purchaseopportunity with the user's EBay account. (Although no EBay link to theproduct is stored in the database, the app can conduct an EBay searchbased on the Name Text data provided from the database.)

If the EBay app decodes the watermark C4FF31 from a National Geographicarticle, it finds that there is no PurchaseAttribute data. It furtherfinds that the associated item is non-purchasable, since the payload isan odd number. So the EBay app then branches to a graceful exit, e.g.,by linking to the National Geographic web site associated with thearticle, per the link in the WebLinkforProduct field of the databaserecord. Or, as noted above, the EBay app can derive a destination in theEBay site, based on metadata provided from the backend database (e.g.,Egyptian-themed items for sale on Ebay).

FIG. 22B shows a flow chart governing operation of the Digimarc Discoverapp. Again, the depicted process begins by checking thePurchaseAttribute data (if any) associated with the watermark payloaddecoded by the app. If there is no such PurchaseAttribute data, the appnext checks whether the item is purchasable (again by reference towhether the payload is even or odd). If it finds the item is notpurchasable (e.g., the watermark is from a National Geographic article),the app responds as in the foregoing paragraph.

(This app may decide—in part—whether to derive a purchase opportunityfrom the metadata, or simply link to an informational page, based on theclue of having purchasing credentials for the user that could be used ina derived purchase opportunity scenario.)

If the Digimarc app finds, instead, that the item is purchasable (e.g.,it senses payload B163A4 from a magazine advertisement for Kleenex AutoPack), it next checks whether the user has specified preferred productfulfillment vendors for such item (e.g., Amazon, EBay, Target, WalMart,etc.). If so, the app queries the user-preferred vendor to determinewhether the Kleenex item is available for purchase. If it finds theKleenex product available from the user-preferred vendor, it readies apurchase transaction for the Kleenex item from that vendor, which theuser can elect to complete or not. (If the item isn't available fromuser-preferred vendor, the app can check second-/third-/etcetera-choicevendors that may have been specified by the user, before resorting to aprogrammed list of still-further alternative vendors that may be tried.)

Returning towards the top of the FIG. 22B flow chart, the Digimarc appmay find that the item indicated by the watermark has a correspondingdatabase record that specifies (in the PurchaseAttribute field) aparticular vendor that should be used in purchasing that item (e.g., asmay be the case if Amazon sponsors a print ad for a Wilson basketball).When the Digimarc app thereby learns that a purchase opportunity shouldbe presented—if at all—from the Amazon web site, the app next checks tosee if user-stored credentials are available for Amazon. If so, the appreadies a purchase transaction for the user's confirmation, using ruledata earlier specified by the user (e.g., sign-in with particular Amazonuser-name and password, and then elect payment by Amazon's One-Clickoption, which is linked to user's VISA card). If no user-storedcredentials are available, the app gracefully exits—such as bypresenting details about the basketball from the Wilson or Amazon websites, or by inviting the user to manually log in to Amazon and completea purchase.

FIG. 24A shows a sample data structure (table) in the memory of theuser's smartphone—detailing preferred vendors for different classes ofitems. For example, if the item corresponding to the watermark is a book(e.g., as determined by the Class data in the FIG. 23 table), then theDigimarc app should first check the Amazon Kindle store for availabilityof the book. If available, the app gives the user the option to purchasewith a single tap of the touchscreen. If unavailable from Kindle, theapp should next check for availability at Half, and then at Abebooks,and then at Amazon (i.e., for a paper version).

The FIG. 24A data structure has different lists of preferred vendors,depending on the item class. For a grocery item, the prioritization maybe Safeway, then Amazon, then EBay. For electronics, the prioritizationmay be Best Buy, then EBay, then Amazon. Etc.

Further information associated with the vendors in the FIG. 24A table isprovided in the FIG. 24B data structure. This table provides the URLs,user credentials, and rule information associated with the variousvendors specified in FIG. 24A.

The password data is depicted in FIG. 24B in cleartext. In actualpractice, it would be better secured against hacking, such as by areference to an entry in an electronic vault, or by information in anelectronic wallet. Similarly, the information for the user's creditcards is desirably not stored in clear text. Instead, the reference,e.g., to Mastercard in FIG. 24B may comprise a tokenized reference to amobile wallet. Such arrangement may store the actual Mastercard numberin the cloud. For some login/password/payment data, FIG. 24B may specifythat the user should be prompted to enter the information as needed.

Information in the tables of FIGS. 24A and 24B needn't all be manuallyentered by the user, in a configuration process. Instead, a process canderive the table data from inspecting available purchasing credentialsand examining the user's past purchasing activities.

Typically, access to the database is subject to a license agreement.This license agreement can contractually require that software using thedatabase operate in accordance with the PurchaseAttribute data, etc.

Of course, the flowcharts depicted in FIGS. 22A and 22B arerepresentative only. Many different arrangements can be employed,depending on the particular needs and circumstances faced by theimplementer. And while described in the context of watermarkinformation, the detailed approaches can be adapted for use with otheridentification technologies—such as barcodes, fingerprinting/featurerecognition, etc. In still other embodiments, such principles can beapplied to audio content—using audio watermarking or fingerprinting.

While the detailed arrangement focused on applications that can be usedfor product purchasing, other applications may not include or desiresuch capability. Instead, such other applications may serve educational,or artistic, or entertainment, or other ends.

Similarly, the party responsible for watermarked imagery may havepreferences as to its use (educational, artistic, entertainment, etc.).

In a more general embodiment, the party responsible for the watermarkedimagery stores metadata, in a backend database, expressingpreferences/limitations/rules concerning actions to be taken based onthe database information. If such a limitation is expressed, theapplication software should respect that requirement. The applicationsoftware, too, may have its own data detailing behaviors that it wantsto enable—both based on preferences of the software provider, and alsobased on the software user. The application software thus follows amultiply-tiered set of rules—first applying any requirements imposed bythe watermark metadata, and then behaving in accordance with its ownrules, further customized by the user-stored information (or derivedfrom historical user behavior data).

In a particular example, backend rule metadata associated with awatermarked photograph in Martha Stewart Living magazine may specifythat the database information is to be used only (1) to post thephotograph to Pinterest, (2) to link to a website corresponding to thearticle in which the photograph is included, or (3) to “Like” thearticle on Facebook.

A Pinterest app encountering such a photograph could naturally pin thephoto to the user's Pinterest account. And a Facebook app could allowthe user to “Like” the article. However, an EBay app could not use thebackend data to initiate a purchase on EBay (although it could, if itits programming allowed, link the user to the website corresponding tothe article).

The metadata of FIG. 23—which includes rule data about item purchasing(i.e., the PurchaseAttribute data), can additionally include rule dataspecifying how the item corresponding to the watermark may be socialized(e.g., by Twitter, and/or by Facebook, and/or by Google+, etc.). If themetadata authorizes socializing the item on Facebook and Google+, butthe user has provided credentials only for Google+, then only thatoption for socialization will be presented to the user.

Although not detailed in the flow charts of FIG. 22A/22B, theWebLinkforProduct data can further serve as a rule that enforces howadditional information about the item should be obtained (e.g., from themanufacturer's site, as opposed to a more general Google search—whichmay lead with commercial advertisements).

The metadata of FIG. 23 is available to all applications. There may beadditional metadata which is private, and made available only to certainsoftware applications. For example, if a photograph in House Beautifulmagazine is watermarked, and is read by a House Beautiful smartphoneapp, that app may be provided metadata in addition to that provided to,e.g., an EBay app. By such private data, the House Beautiful app canenable watermark-responsive behaviors that other applications cannotprovide.

(More information on use of watermark metadata and its uses is found inpublished patent application 20070156726.)

Printed Response Codes

In accordance with another aspect of the technology, a response code isincluded in printed content, such as advertising, newspaper/magazineeditorial content, product packaging, etc. However, unlike knownresponse codes (e.g., QR codes), the present code includes semanticinformation for human viewers. Such information can include, e.g., logosfor social networking services—informing viewers as to the action(s)that the code can launch. While providing various benefits to thereaders, use of such codes also leads to electronic gathering of usagemetrics for print publications—a feature generally unavailable to printpublishers.

FIG. 25 shows the first page of an article of a magazine articleentitled “Answering the Trickiest Questions,” which addresses the topicof discussing difficult issues with children. In the lower corner is anexemplary response code 251 showing a particular form of implementation.The code is shown in larger form in FIG. 26. A variant form is shown inFIG. 27. (The additional logo in FIG. 27 indicates entry of commentsabout the article.)

As can be seen, the response code is not the uninformative black andwrite gridded pattern of a conventional machine readable code. Instead,the code includes text that the viewer can read (“Scan here to sharethis article”), and graphics that the viewer can recognize as logos ofFacebook, Twitter, Pinterest, Google+ and email.

Not apparent to the viewer, however, is that the code also conveys asteganographic digital watermark. When sensed by a suitable smartphoneapp, the watermark causes the smartphone to load an HTML landing webpage—such as that shown by FIG. 29.

An interstitial advertisement page, such as shown by FIG. 28, may bepresented while the landing page loads (or, if the landing page loadsfaster than, e.g., 1.5 seconds, then for a longer period specified bythe publisher). The interstitial page has the same header as the landingpage, providing a more seamless transition between FIGS. 28 and 29 tothe viewer. This header includes both the brand of the magazine, and thetitle of the magazine article from which the code was scanned. In someembodiments the interstitial page is static—not clickable (e.g., sayingsimply “This share is brought to you by <sponsor, e.g., GoodHousekeeping>”)—to make sure the reader isn't side-tracked from reachingthe FIG. 29 landing web page.

The FIG. 29 landing web page presents UI buttons that are selectable bythe reader to share an online version of the magazine's “Answering theTrickiest Questions” article across multiple social networks. Otheroptions are also available, including “liking” the article on Facebook,Twitter and Google+, pinning the article's artwork to Pinterest, sendingan email to a friend with a link to the online copy of the article, andentering comments for presentation with the online copy of the article.

Selecting one of the “Share” buttons from the FIG. 29 page establishesthe same connection to the chosen social network user interface as if a“Share” icon for that network had been clicked from the publisher'sarticle web page. FIGS. 30 and 31 show an illustrative sequence ofscreens if the Twitter/Share button is selected. The first screen allowsthe reader to sign-in to Twitter. The second screen allows the reader toedit and send a Tweet containing the article link. (The FIG. 30 screenis skipped if the reader is already signed-into Twitter.)

If the reader's smartphone has a specialized app installed for theselected social network (e.g., a Facebook mobile app), then that app maybe launched when the link is posted to that network.

Whenever the reader shares an article link using the detailedtechnology, information in addition to the URL can be provided. The nameof the person, the name of the enabling service, and the name of theprint magazine, can all be included in the shared information (e.g.,“Linked by Alice Smith, via Sharemarc™, from Parents Magazine”).

Selecting one of the “Follow” buttons from the FIG. 29 page leads to thepublisher's page on the selected social network. FIG. 32 shows theresult if the Pinterest/Follow button is selected: the Good Housekeepingmagazine page on Pinterest loads. (In a variant embodiment, each articlemay have its own social network page—complete with the article text andassociated comments. In still another embodiment, clicking the “Follow”button leads to the social network account of the article's author.)

Selecting a “Comment” button can trigger different actions, depending onimplementation.

One implementation presents a new UI on the smartphone—with the sameheader as the previous pages, but inviting the reader to tap in acentral area to enter a comment text (using an on-screen keyboardappears, as is familiar)—optionally with a user name/password. When thereader finishes typing the comment, and taps a Post button, the commentis sent to the magazine's web site, where it is added to public commentspresented at the end of the online article. The comment entry UIdisappears, and the smartphone next displays the comment section of theonline article page, where the comment will soon appear.

Another implementation responds to the reader's tap of the “Comment”button by loading a web page displaying the online article, scrolledtowards the end to the online form where public comments are entered.The reader can then review other readers' comments, and interact withthe publisher's existing comment form to enter their own comments.

Some magazines provide a mobile Facebook comments app. This app can belaunched by a tap to the FIG. 29 Comments button, to allow the reader toenter an article comment.

If the reader selects the Email button from FIG. 29, the reader'spreferred email app opens, with a message draft that includes a link tothe online article—poised for the reader to type an email address andsend.

It will be recognized that the landing page shown in FIG. 29 isillustrative only. In other implementations, other layouts can be used,with more or less social networks, more or less sharing options, etc. Insome implementations, all the landing pages for a particular magazine,or for a particular magazine issue, are actually the same page template,customized dynamically (e.g., using HTML5) to correspond to theparticular response code involved. One particular approach has a simpleconfiguration template associated with each issue, which identifies thearticle name, page number, machine readable code, online link URL, etc.

The codes shown in FIGS. 26 and 27 are similarly illustrative only;countless variations are possible (some devoid of social network icons).Desirably, however, the shape and graphic content of the codes aregenerally consistent within the magazine, and preferably are consistentbetween different magazines and even different publishers—to aid inpublic recognition. (Color might be changed as necessary, e.g., toconform to the magazine's signature colors, or for better presentationon the article's background color.)

As shown in FIG. 25, the preferred code is small enough to be easilyplaced on a printed page, but large enough to be noticeable by thereader. The size of the code is typically less than 2 inches in itsmaximum dimension, and may be less than 1.5, 1.1, 0.8, 0.6 or 0.4 inchesin that dimension. The other dimension is smaller, such as by a factorof 2, 3, 4 or more. The depicted code is about 1.75 by 0.65 inches.Typically, the code is non-square, but it needn't be rectangular. Forexample, codes with one or more curves edges can be used.

Some implementations provide a reader history option than isUI-selectable by the reader from one or more of the app screens, torecall previous magazine code reads and shares. These can be organized,at the reader's election, by date, by magazine title, by article title,by network, etc. By recalling such history, the user can return toarticles that were earlier of interest, and examine new comments, sharewith new friends, etc.

The above-described functionality can be provided by magazine-brandingof a generic watermark-reading smartphone application. As anotheroption, a watermark reader can be integrated into a magazine's existingmobile application.

The technology provides the publisher with a variety of real-time,online analytic reports and charts. These detail, for example, thenumber of times the printed code in each article was scanned, the numberof times the article link was shared on each of the social networks, thetraffic driven to the publisher from such sharing, the total shares andtraffic for each article, the total shares and traffic for each network,the total shares and traffic for each article/network combination,details of the foregoing activity by geographic areas and by date/hour,and aggregate counterparts to the foregoing across all networks andacross all articles in a particular print issue, all issues in amagazine brand, and all magazine brands owned by a publisher.

(“Shares” in the foregoing refers to any event in which a reader tapsone of the options on the landing page (e.g., Like, Share, Follow,Recommend, Email, Comment), and completes the action from the socialnetwork site. “Traffic” refers to events in which the reader clicks tolink to the online article.)

Alternatively, or additionally, analytics available from Google can beemployed. Similarly, analytics from the social networks (e.g., providedby the Facebook Open Graph API) can be used. These latter analyticsallow tracking of, e.g., how many friends (in aggregate) were exposed toa shared link, how many people shared the article on the social network,how many of their friends were exposed to it, etc.

In this era that some decry as witnessing the demise of print media, thedetailed technology provides many benefits to print publishers—amongthem: new advertising revenue streams, and exposure—by sharing—to newaudience members (potentially leading to new print subscriptions).Moreover, while the print subscriber base for most magazine farout-numbers the digital reader base, the print side of the business hasno detailed analytics to measure consumption of the content. The presenttechnology provides a sampling of such previously-unavailablestatistics. Such information can help publishers decide the type ofcontent and coverage angles that should be pursued.

Other Arrangements

Still another application of the present technology is an image-basednews feed. Pinterest or other social network can examine the geographiclocations of members who are actively posting images, and identify thoselocations from which imagery is being posted at rates higher thanstatistical norms. Images posted by users in such location can berandomly selected (perhaps after some brief quality assurance metrics,such as sharpness analysis) and output to an image feed that other userscan tune to and review. Such feed of imagery can be distributed bychannels and means other than the social network(s) from which theimages originated. For example, the image feed may be converted into anMPEG stream and distributed by YouTube and other video distributionservices.

Instagram is a social network whose popularity resides, in part, on theeasy application of filters to smartphone-captured images, to give themartsy effects. (The filters include X-Pro II, Lomo-fi, Earlybird, Sutro,Toaster, Inkwell, Walden, Hefe, Apollo, Poprocket, Nashville, Gotham,1977, and Lord Kelvin. One converts the image colors to sepia; anotheradds a peripheral blurring vignette, etc.) Desirably, some or all ofthese filters also overlay a digital watermark pattern on the image.This watermark can encode an identifier of the user (e.g., theirInstagram login, or their email address, etc.). That way, if an image islater repurposed or otherwise used in an unforeseen way, it can still beassociated back with its the originating user.

Another technology involves smartphone-captured video. A software appexamines the sequence of frames and selects representative keyframes. (Avariety of techniques for keyframe selection are known. See, e.g., U.S.Pat. Nos. 5,995,095 and 6,185,363, and published application20020028026.) The app presents these keyframes to the user, who selectsone (or more). A user-selected filter is then applied to the image(s),e.g., in the manner of Instagram. Again, a watermark is added with thefilter. The image (which may be termed a Photeo) is then uploaded to asocial networking service, together with some or all of the video. Bothare stored, and the image is presented with other images in the user'saccount (e.g., on a pinboard, etc.), from which it may be re-posted bythe user or others to other locations, etc. When any copy of the imageis later selected (e.g., tapped or clicked) by a viewer, the embeddedwatermark is desirably decoded and links to the full video, which isthen rendered to the viewer.

In such arrangement, the image can be enjoyed just like any other image,but it also provides the added element of a video clip if someone sochooses to watch. Such functionality also persists through printing.That is, if the image is printed, and the print document is thereafterphotographed by a smartphone, an app on the smartphone can again readthe watermark and launch the linked video into playback. Postcards andprint booklets can thus become memento portals through which people canexperience video captured at weddings, parties, concerts, etc.

Other arrangements can involve both watermarking and fingerprinting.

Consider a magazine or newspaper article that is digitally watermarked(e.g., by background speckling, or by watermarking of an included image,etc.) and enrolled in the Digimarc Discover service (detailed, e.g., inapplication 20120284012). This service receives watermark payload datadecoded by a Digimarc Discover app from a printed publication, andidentifies a corresponding response that should be triggered by thatwatermark (e.g., linking to an online version of the article.)

The text of the watermarked article is obtained, and is enrolled in afingerprint database—along with metadata for the article (e.g., title,author, publication name and date, etc.). This text can be obtained byloading it from a URL supplied by the publisher, which points to thetext at its web site. (The URL may be indicated by the watermark payloaddata.) Alternatively, the text may be obtained from anotherrepository—such as from the Digimarc Discover back end server.

The fingerprint can comprise scale invariant features (or other robustgraphic features) of the text as laid out in a graphic representation ofthe distributed article. Alternatively, the fingerprint can comprisesnippets of text (OCR'd if necessary) that can serve as quasi-uniqueidentifiers for the article.

A web crawler then examines computer sites on the internet, using thefingerprint data and/or the associated metadata, to identify instancesof where the article has been shared. Social networks, such as Facebook,can be checked for references to the article shared among members, usingthe Facebook API.

Demographic information is gleaned from the locations where the articleor its associated metadata are found. For example, a particular articlein House Beautiful may be discovered to be popular among women 14-18years old, in the US Midwest. Another article may be posted to a website that draws most of its traffic from residents of Jordan. The numberof comments posted about the article, and sentiment analysis of same,provide additional information about the article's popularity andaudience reaction.

(Although described in the context of printed articles, the sameapproach is useful with other media types.)

Other Comments

FIGS. 18-21 illustrate certain of the other aspects of the presenttechnology.

The assignee's pending patent application Ser. No. 13/149,334, filed May31, 2011, Ser. No. 13/174,258, filed Jun. 30, 2011, and Ser. No.13/425,339, filed Mar. 20, 2012, and published application 20100228632,detail technologies that are related to the presently-describedtechnologies.

While certain arrangements involve imagery captured with a user'ssmartphone camera, this is not essential. For example, implementationsof the present technology can utilize image data obtained otherwise,such as electronically transmitted from another source (e.g., a friend'ssmartphone), or obtained from the web.

Likewise, while software on the smartphone typically performs extractionof identification data from the image data in certain of the detailedarrangements, this, too, is not essential. For example, the phone cansend the image data to a processor remote from the phone (e.g., at thesocial networking service or in the cloud), which can perform extractionof the image identification data. Or the extraction can be distributed,with initial phases of the process (e.g., filtering, and/or FFTtransforming) performed on the handset, and latter phases performed inthe cloud.

The focus of this disclosure has been on still imagery. But it will berecognized that the detailed technologies can likewise be employed withvideo and audio content.

Similarly, while many of the particular embodiments were described inconnection with Pinterest, it will be recognized that such technologiescan likewise be used in connection with other social networks.

It should be understood that features and arrangements detailed inconnection with one embodiment can likewise be incorporated into otherembodiments. Such combinations and permutations are not exhaustivelydetailed, as their implementation is straightforward to theartisan—based on this disclosure.

As noted earlier, the technology also finds application with barcodes. Abarcode can be photographed with a smartphone, and its payload data canbe decoded (either locally at the smartphone, or sent to anothercomputer for decoding). The decoded barcode data is then used to accessa product database to obtain information about the associated product.Once the product has been identified, the other detailed methods can beutilized.

While some embodiments involve watermarking each image in a magazinewith a different watermark payload, this is not necessary. For example,all the images on a single page—or in a single article—can bewatermarked with the same payload. When any of them is imaged by theuser's smartphone, the smartphone app may recall pristine versions ofall images on the page (or in the article), and present them as agallery of thumbnails, from which the user can select the desired image.In some magazine articles, only a single image may be watermarked oneach page, yet its capture by the smartphone camera will result in thesystem presenting thumbnails of all images on that page (or in thatarticle).

In embodiments in which multiple related images are presented to theuser, they needn't all be presented on a single screen (as in a gridlayout). In other arrangements, a single image may be presentedper-screen, and the user interface can allow transitioning to otherimages by swiping a finger across the screen, or by touching a Nextbutton or the like.

While occasional reference has been made to layout, this term does notnarrowly refer only to physical placement of one item (e.g., an image)relative to another on a page. Rather, it also encompasses aspects ofstructure, sequence and organization—reflecting associations betweenitems, and the experiential flow towards which the publication subtly orovertly guides the user in navigating the publication.

Similarly, references to images, photos, and the like should not benarrowly construed. For example, a photo refers not just to a paperprint of a picture, but also to the set of digital data by which thepicture is represented.

Likewise, while reference has been made to “posting” images to a socialnetworking service, it will be recognized that such posting need notinvolve transfer of image data. Instead, for example, the posting maycomprise providing address data for the image—such as a URL. Postingencompasses the act of the user in directing the posting, and the act ofthe social networking service in effecting the posting. Many socialnetworking services provide APIs to facilitate posting of data to theservice using third party software.

Naturally, when reference is made to an item in the singular (e.g., “aphoto”), it will be recognized that such description also encompassesthe case of plural items.

When an image is identified, metadata associated with the image may beaccessed to determine whether re-distribution of the image if permitted.Some image proprietors are eager to have their imagery re-distributed(e.g., promotional imagery for commercial products); others are not. Thesmartphone app can vary its behavior in accordance with such metadata.For example, if the metadata indicates that redistribution is notpermitted, this fact may be relayed to the user. The software may thenpresent alternate imagery that is similar (e.g., in subject matter orappearance) but is authorized for redistribution.

It will be recognized that product-identifying information may bedetermined from sources other than image data. For example, a productidentifier may be read from an NFC (RFID) chip on a product, on a shelfdisplay, or elsewhere, using the NFC reader provided in manysmartphones.

Still further, acoustic identification may sometimes be used (e.g.,ultrasonic signals, such as are used to identify retail stores to theShopKick app). In such arrangements, a user's phone may detect a uniqueultrasonic signature that is present in a home furnishings aisle atMacy's department store. The smartphone can use this information todetermine the phone's location, which is then provided to a databasethat associates such location information with collections of imagesdepicting nearby merchandise. These images may be presented on thesmartphone display to the user, who may then elect to like one or moreof these products, or post one or more of the images (or related images,discovered as described above) to the user's social network account asdescribed above. (Shopkick's technology is further detailed in patentpublication 20110029370.)

While reference was made to app software on a smartphone that performscertain of the detailed functionality, in other embodiments thesefunctions can naturally be performed otherwise—including by operatingsystem software on the smartphone, by a server at a social networkingservice, by another smartphone or computer device, distributed betweensuch devices, etc.

While reference has been made to smart phones, it will be recognizedthat this technology finds utility with all manner of devices—bothportable and fixed. PDAs, organizers, portable music players, desktopcomputers, laptop computers, tablet computers, netbooks, wearablecomputers, servers, etc., can all make use of the principles detailedherein. Particularly contemplated smart phones include the Apple iPhone4s, and smart phones following Google's Android specification (e.g., theMotorola Droid 4 phone). The term “smart phone” should be construed toencompass all such devices, even those that are not strictly-speakingcellular, nor telephones (e.g., the Apple iPad device).

(Details of the iPhone, including its touch interface, are provided inApple's published patent application 20080174570.)

While many of the illustrative embodiments made reference to digitalwatermarking for content identification, in most instancesfingerprint-based content identification can be used instead.

The techniques of digital watermarking are presumed to be familiar tothe artisan. Examples are detailed, e.g., in Digimarc's U.S. Pat. No.6,590,996 and in published application 20100150434. Similarly,fingerprint-based content identification techniques are well known.SIFT, SURF, ORB and CONGAS are some of the most popular algorithms.(SIFT, SURF and ORB are each implemented in the popular OpenCV softwarelibrary, e.g., version 2.3.1. CONGAS is used by Google Goggles for thatproduct's image recognition service, and is detailed, e.g., in Neven etal, “Image Recognition with an Adiabatic Quantum Computer I. Mapping toQuadratic Unconstrained Binary Optimization,” Arxiv preprintarXiv:0804.4457, 2008.) Use of such technologies to obtainobject-related metadata is likewise familiar to artisans and isdetailed, e.g., in the assignee's patent publication 20070156726, aswell as in publications 20120008821 (Videosurf), 20110289532 (Vobile),20110264700 (Microsoft), 20110125735 (Google), 20100211794 and20090285492 (both Yahoo!).

Linking from watermarks (or other identifiers) to corresponding onlinepayoffs is detailed, e.g., in Digimarc's U.S. Pat. Nos. 6,947,571 and7,206,820.

Additional work concerning social networks is detailed in Digimarc'spatent application Ser. No. 13/425,339, filed Mar. 20, 2012.

The camera-based arrangements detailed herein can be implemented usingface-worn apparatus, such as augmented reality (AR) glasses. Suchglasses include display technology by which computer information can beviewed by the user—either overlaid on the scene in front of the user, orblocking that scene. Virtual reality goggles are an example of suchapparatus. Exemplary technology is detailed in U.S. Pat. No. 7,397,607and 20050195128. Commercial offerings include the Vuzix iWear VR920, theNaturalpoint Trackir 5, and the ezVision X4 Video Glasses by ezGear. Anupcoming alternative is AR contact lenses. Such technology is detailed,e.g., in patent document 20090189830 and in Parviz, Augmented Reality ina Contact Lens, IEEE Spectrum, September, 2009. Some or all such devicesmay communicate, e.g., wirelessly, with other computing devices (carriedby the user or otherwise), or they can include self-contained processingcapability. Likewise, they may incorporate other features known fromexisting smart phones and patent documents, including electroniccompass, accelerometers, gyroscopes, camera(s), projector(s), GPS, etc.

The design of smart phones and other computer devices referenced in thisdisclosure is familiar to the artisan. In general terms, each includesone or more processors (e.g., of an Intel, AMD or ARM variety), one ormore memories (e.g. RAM), storage (e.g., a disk or flash memory), a userinterface (which may include, e.g., a keypad, a TFT LCD or OLED displayscreen, touch or other gesture sensors, a camera or other opticalsensor, a compass sensor, a 3D magnetometer, a 3-axis accelerometer,3-axis gyroscopes, a microphone, etc., together with softwareinstructions for providing a graphical user interface), interconnectionsbetween these elements (e.g., buses), and an interface for communicatingwith other devices (which may be wireless, such as GSM, CDMA, 4G,W-CDMA, CDMA2000, TDMA, EV-DO, HSDPA, WiFi, WiMax, mesh networks, Zigbeeand other 802.15 arrangements, or Bluetooth, and/or wired, such asthrough an Ethernet local area network, a T-1 internet connection, etc).

More generally, the processes and system components detailed in thisspecification may be implemented as instructions for computing devices,including general purpose processor instructions for a variety ofprogrammable processors, including microprocessors, graphics processingunits, digital signal processors, etc. These instructions may beimplemented as software, firmware, etc. These instructions can also beimplemented to various forms of processor circuitry, includingprogrammable logic devices, FPGAs, FPOAs, and application specificcircuits—including digital, analog and mixed analog/digital circuitry.Execution of the instructions can be distributed among processors and/ormade parallel across processors within a device or across a network ofdevices. Transformation of content signal data may also be distributedamong different processor and memory devices.

Software instructions for implementing the detailed functionality can bereadily authored by artisans, from the descriptions provided herein,e.g., written in C, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme,Ruby, etc. Mobile devices according to the present technology caninclude software modules for performing the different functions andacts.

Commonly, each device includes operating system software that providesinterfaces to hardware resources and general purpose functions, and alsoincludes application software which can be selectively invoked toperform particular tasks desired by a user. Known browser software,communications software, photography apps, and media processing softwarecan be adapted for many of the uses detailed herein. Software andhardware configuration data/instructions are commonly stored asinstructions in one or more data structures conveyed by tangible media,such as magnetic or optical discs, memory cards, ROM, etc., which may beaccessed across a network. Some embodiments may be implemented asembedded systems—a special purpose computer system in which theoperating system software and the application software isindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

In the interest of conciseness, the myriad variations and combinationsof the described technology are not cataloged in this document.Applicants recognize and intend that the concepts of this specificationcan be combined, substituted and interchanged—both among and betweenthemselves, as well as with those known from the cited prior art.Moreover, it will be recognized that the detailed technology can beincluded with other technologies—current and upcoming—to advantageouseffect.

To provide a comprehensive disclosure, while complying with thestatutory requirement of conciseness, applicantsincorporate-by-reference each of the documents referenced herein. (Suchmaterials are incorporated in their entireties, even if cited above inconnection with specific of their teachings.) These references disclosetechnologies and teachings that can be incorporated into thearrangements detailed herein, and into which the technologies andteachings detailed herein can be incorporated. The reader is presumed tobe familiar with such prior work.

1. A method comprising: receiving first imagery captured by a smartphonecamera, the first imagery representing a first retail product located ata retail location, and presenting the first imagery on a screen of thesmartphone; providing the first imagery to a processor to producefingerprint data therefrom, the fingerprint data being utilized toidentify the first retail product; receiving second imagery representinga second retail product, identified as a product recommendationassociated with the first retail product, the second imagery beingsourced from a source different than the smartphone camera; presenting,on the screen of the smartphone, the second imagery; receiving userinput via a touch screen of the smartphone; as a consequence of saiduser input, initiating an action.
 2. The method of claim 1 in which saidaction comprises initiating an online posting of the first imagery orthe second imagery for viewing by others.
 3. The method of claim 1 inwhich the action comprises initiating an online purchase of the secondretail item.
 4. The method of claim 1 in which the action comprises arequest for directions within the retail location to a location at whichthe second retail product is located.
 5. The method of claim 4 in whichthe directions are determined, at least in part, based on a radiofrequency signals emitted from the smartphone.
 6. The method of claim 1in which the action comprises a request for pricing of the second retailproduct from multiple, different sources.
 7. The method of claim 1 inwhich the second retail product is identified, at least in part, basedon information associated with a user of the smartphone.
 8. The methodof claim 1 further comprising: receiving third imagery, the thirdimagery being sourced from a source different than the smartphonecamera, in which the third imagery comprises one or more alternativeimages of the first retail product.
 9. The method of claim 8 in whichthe action comprises initiating an online posting of the one or morealternative images for viewing by others.
 10. A smartphone comprising: atouchscreen display; a camera for capturing imagery; one or moreelectronic processors programmed for: handling first imagery captured bythe camera, the first imagery representing a first retail productlocated at a retail location; controlling presentation of the firstimagery on the touchscreen display; providing the first imagery to aprocessor to produce fingerprint data therefrom, the fingerprint databeing utilized to identify the first retail product; controlling receiptof second imagery representing a second retail product, identified as aproduct recommendation associated with the first retail product, thesecond imagery being sourced from a source different than saidsmartphone camera; controlling presentation of the second imagery on thetouchscreen display; receiving user input entered via the touchscreendisplay; as a consequence of said user input, initiating an action. 11.The smartphone of claim 10 in which said action comprises initiating anonline posting of the first imagery or the second imagery for viewing byothers.
 12. The smartphone of claim 10 in which the action comprisesinitiating an online purchase of the second retail item.
 13. Thesmartphone of claim 10 in which the action comprises a request fordirections within the retail location to a location at which the secondretail product is located.
 14. The smartphone of claim 13 in which thedirections are determined, at least in part, based on a radio frequencysignals emitted from the smartphone.
 15. The smartphone of claim 10 inwhich the action comprises a request for pricing of the second retailproduct from multiple, different sources.
 16. The smartphone of claim 10in which the second retail product is identified, at least in part,based on information associated with a user of the smartphone.
 17. Thesmartphone of claim 10 in which the one or more electronic processor areprogrammed for: controlling receipt of third imagery, the third imagerybeing sourced from a source different than the smartphone camera, inwhich the third imagery comprises one or more alternative images of thefirst retail product.
 18. The smartphone of claim 17 in which the actioncomprises initiating an online posting of the one or more alternativeimages for viewing by others.